4 Million credits!!

Not much of interest been happening on my Boinc account, the hardware is all behaving and now that summer is here, I do turn everything (including the server) off over the weekends as the air-conditioning doesn't run unless there is someone in the building.
Have just reached 4 million credits, mainly thanks to concentrating on Collatz and Milkyway.

Still alive and kicking ...

Despite all appearances, I am still crunching. Collatz and Milkyway haven't been giving out work on a reliable basis and due to temperatures in Sydney getting over 41'C over the weekend, the credit accumulation has been somewhat impaired. I have a trade show to attend this week, so I will spend some time getting the GPUs off Seti@Home next week.

Collatz Conjecture user of the day!

I am the Collatz user of the day for the 9th November :-)

3's 3G ... stable but S-L-O-W!

Due to the problems I have been having with Unwired and their lack of a consistent service, I got a 3G modem from 3.
It is far more stable than Unwired but the speed in Dee Why seems to get to a ripping 12Kb/s ... and thats it.

I did get an interesting comment from Jonathan about Intel's Progress through Processors and their relationship with GridRepublic. His comment is at the bottom of my Oct 24th post.

Karmic Koala cont...

I really don't think this is Ubuntu's best efforts, there are way too many bugs in the new release at this point. I am getting at least one crash per hour at the moment. They released an update over the weekend that seems to have sorted out the client side of Ubuntu One's cloud "thingy" ... but there is still lots more. The one that troubles me most is the ECC memory error from the kernel.

Upgrade to Ubuntu's Karmic Koala

I just couldn't resist and decided to upgrade a perfect/tuned/optimized machine to the new version of Ubuntu via the Synaptic distribution upgrade. All went well for half an hour or so and then the upgrade crashed. A whole barrage of bug reporting options appeared and it eventually landed up sitting there with the normal desktop showing. I knew the kernel had upgraded etc, so I rebooted.
The results were better than I expected, it booted and even BOINC was alive and still running the 190 series driver from the old installation. However I keep getting warnings about my BIOS and ECC memory problems and needing to reboot. Looking at the forums, I am not the only one affected by this. I will give it a few days and see if there are any patches released ... if not, I will take the plunge and do a clean install.
The only BOINC causality appears to be QMC@Home. The Orca work-units appear to run OK (and as slowly as usual) but the normal workunits crash due to libstdc++.so.5 missing from Karmic. I will try and drag them out from an older distro when I have some time.
My initial feeling is that the upgrade from 9.04 (Jaunty) wasn't really worth the effort but will see if that feeling changes when a few more bugs are ironed out.

Intel's Progress Through Processors in Facebook

I have seen a little ad on my Facebook page for a while for Intel's "Progress Through Processors" application in some sort of partnership with GridRepublic and decided to have a look at it due to my interest in volunteer computing. Well ... err, it's Boinc by a different name! I'm not sure what Intel is trying to do with this. I'm pretty sure Berkeley was the driving force behind this and not Intel?! This application has even created a different credit system of "G-hours" instead of "Cobblestones"

I currently have completed 734,242 G-hours for what it is worth...

They also seem to think I have an account with NanoHive, a project I have never signed up for.

I guess the up-side is that the projects will get a lot more "non-technical" crunchers signing up now that it is on Facebook but I give a "thumbs down" to Intel for trying to make it their own idea. Give credit where it is due!

R.I.P. Unwired in Sydney

When I moved into my current apartment around a year ago, I didn't want to commit to a 12/24 month ADSL contract and at that time the Aussie pre-paid 3G offerings were not worth considering. I opted for Unwired and after a couple of days of moving their "wabbit" modem around, I eventually got an "Ok" service by taping it to the inside of my lounge window and using a Wi-fi router. The metal tray in the picture is there to try and stop it hunting between the two antennas in the modem as that disrupts any data through-put.

This arrangement has done the job and has even allowed the occasional VOIP call when the humidity/tides/moon phase etc have been just right for an optimum connection.

In the last week however, the service has diminished to a point where it isn't usable anymore. The modem shows "connecting..." for anything between 5 minutes and 5 hours before allowing a couple of minutes of traffic.

I'm not sure what the problem is but am moderately sure nothing has changed on my side. For the few brief minutes per day when I get service, the signal strength is what it normally was.

So, anyone on the northern beaches who is considering Unwired ... DON'T! I will go shopping on the weekend and see what I can find in the way of 3G modems and reasonably priced plans.

3 Million :-)

Well at long last, the 3 million mark has been reached! Its a bit depressing when you see some users doing that in "a day or two" but this is my contribution since I joined SETI in 1999.

Collatz for 64-bit Linux CUDA

On the Collatz website, there is a section called "optimized applications" with a 64-bit Linux CUDA client. I used the exact same procedure to install it as is listed for the VLAR-kill SETI application a bit earlier in this blog. I'm knocking out a work-unit in about 33 minutes on the 9800GT card ... and each one is worth around 570 cobblestones, so not a bad return.
I'm still not sure what was causing the GPUGrid units to fail on this card but I am hoping to keep it running on a mixture of Collatz and SETI. At some stage I may look at building an app_info.xml that allows me to run both CPU & GPU work units for Milkyway on this machine. The only problem at the moment is that they will only give me 12 work-units which the CPU can kill off on its own in a day. I have already got the 8600GT/Celeron running a both SSE3 and a CUDA app for Milkyway and I am always out of work units.

Linux nVidia driver update

I do get a lot more credit per hour on GPUGrid than I do on SETI, so I looked for another project for the 9800GT card now that I have problems with GPUGrid. I attached to Collatz but that is running a beta application for Linux Cuda and despite spending an hour messing with the set-up and moving files around, I couldn't get it to work. It trashes work units immediately each time. The Windoze version seems to run fine on the 'ol 8600GT.
One of the comments on the Collatz message board says you need driver version 190.xx ...so, I upgraded to 190.32. It was a painless upgrade and everything else seems to work fine except Collatz. I have allowed new work units from GPUGrid and will watch to see if they complete without errors. If not ... then I am back to the joys of trying to keep a 9800GT card running!

Travelling the same road again...

Well, the Leadtek 9800GT ran well for the first 3 weeks and now, as with the Galaxy card, there are more and more work units crashing. The last three GPUGrid work units have returned a "process exited with code 1 (0x1, -255)". The Boinc wiki says that this could be caused by out-of-date drivers but I don't buy that as things were working really well last week.
Anyway, will move the card over to SETI Cuda work units and see if they also report errors.

Suckered ... again!

I decided to upgrade the nVidia driver on the Celeron/8600GT/Windoze machine to the latest available to be able to run GPUGrid and Einstein on this machine. So, I download the driver and install it, a reboot follows ... and no CUDA task starts up. They are all there with "Ready to Start" next to them but nothing is running. After a minute or two of messing around with no success, I decide that the driver I have installed hasn't installed properly. I uninstall and install an older driver that I have used before ... no luck. So I uninstall again and after all the reboots, go back to the original 182.50 version. Still no luck. I then upgrade to the latest BOINC client, still nothing. At this stage I resort to Google and find this little gem : "If every cpu is running in "high priority mode" the cuda task are switched to "waiting to run""

Aborted a QMC task that was busy and all is back to normal!

In the words of Homer Simpson ... "D'oh!"

One more thing...

I set about upgrading my home Ubuntu 64-bit PC in the manner I listed below and it took me some time to realize why SETI kept trying to download its stock CPU application. When I copied the files into the project directory, I hadn't checked the permissions and it turns out that they were only visible to 'root'.
I should have picked that up much sooner from the fact that the Boinc Manager wasn't picking up the "app_info.xml" file ...
Anyway, the Leadtek 9400GT is also crunching on CUDA2.2 now!

CUDA 2.2 VLAR Kill SETI application for Ubuntu??

Well, there is still no CUDA application for Linux from SETI and ever since I started with a nVidia card, I have been using the application from Crunch3r. There are two little issues with this application, the first is that it took around 100% of one of my CPU cores to run and secondly, VLAR (Very Low Angle Range) work units ran and took ages to complete.
While trolling the internet at lunch time, I came across a forum post on Crunch3r's site that mentions a Linux application with a VLAR Kill option. I have been using the Windoze equivalent and decided to pursue this.
A bit more reading revealed that the CUDA 2.2 version uses a lot less CPU time than the original application.

So, here is how I got it to work (there are probably easier/cleaner ways of doing it, but this is how I got to the end result).

First, you need the CUDA 2.2 libraries. I got them from the Nvidia site (http://www.nvidia.com/object/cuda_get.html). They don't list a CUDA 2.2 toolkit for 64-bit Ubuntu 9.04, so I download the one for Ubuntu 8.10.
This gives you a ".run" file which confused me a bit to start with. I got it installed with the following:

Navigate to the directory of the .run file. I had mine on the desktop so I used "cd ~/Desktop".
Then "chmod +x cudatoolkitxxx.run"
And "./cudatoolkitxxx.run"
I just used the default installation directories.

The next stage is to copy the required files from the CUDA Toolkit installation into the projects/setiathome.berkeley.edu directory. You will need:

libcudart.so
libcudart.so.2
libcudart.so.2.2
libcufft.so
libcufft.so.2
libcufft.so.2.2

The actual application comes from a link on Crunch3r's site (http://calbe.dw70.de/mb/viewtopic.php?f=9&t=116). I used setiathome-CUDA_2.2_6.08.x86_64_vlarkill.

Here is one part that I'm not sure about but it seems to be necessary. You need to copy the application to /usr/bin as well as projects/setiathome.berkeley.edu .This is a prime example of where there are better ways of doing things!

You will need an app_info.xml file in projects/setiathome.berkeley.edu to get it all to work. What I did here was take the included file from Crunch3r's normal CUDA app (http://calbe.dw70.de/linux64.html) and edit it, changing the name of the application to match setiathome-CUDA_2.2_6.08.x86_64_vlarkill. I left the names of the CUDA library files as they were.
Once the app_info.xml file was in the correct directory, I restarted the BOINC client and amazingly it all seems to work fine with a CPU usage rate of 2%. This means I have almost a whole additional core to run CPU work units. This is still using the 6.4.5 Boinc client

Credit to Crunch3r and sunu (on KWSN forums) for the info they have posted that got me through this!

Upgrading nVidia drivers in Ubuntu to ver 185.18.36

All seems to be going well with the Leadtek 9800GT so far, however GPUGrid work units started crashing at a good rate of knots this morning. Their home page carries the ominous phrase "The new Linux application is now based on CUDA2.2 and requires drivers 185.xx."
Up until now I have been using the 180.x driver that is in the Ubuntu repositories ... but alas, nothing newer than that available.

I resorted to using the following to get it to work (all entered into a terminal window):

sudo sh -c "echo 'deb http://ppa.launchpad.net/nvidia-vdpau/ppa/ubuntu jaunty main' >> /etc/apt/sources.list"
sudo sh -c "echo 'deb-src http://ppa.launchpad.net/nvidia-vdpau/ppa/ubuntu jaunty main' >> /etc/apt/sources.list"

Followed by (for GPG key):

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CEC06767

And finally:

sudo apt-get update && sudo apt-get install nvidia-185-modaliases nvidia-glx-185

Everything seemed to work fine after a reboot, except for the Boinc client trashing all the previously downloaded SETI Cuda work units. It did, however download 3 new GPUGrid work units and seems to be crunching merrily.

So far, so good ...

The Leadtek 9800GT has been running for a good couple of hours now (it was turned off over the weekend however) and all seems to be good. It is doing a SETI work unit in about 40 minutes (actual wall time) and seems to have a steady temperature of 70'C which is pretty good as there is no case fan in this machine yet.
The Galaxy was also stable for the first couple of days before it started becoming unreliable, so I won't hold my breath just yet.

Leadtek 9800GT

Well, a Leadtek 9800GT arrived at the office via a courier yesterday afternoon. There are several differences between this card and the original Galaxy. The first and most obvious is that the Leadtek doesn't have plastic cowling that allows hot air to be expelled towards the DVI plugs and out of the case (this may present a problem in summer). The other difference that has me a bit confused is that the Leadtek doesn't require the 6-pin power cord that most the faster cards do. I'm wondering if this means it is a low power version.

I have elected to install it in my AMD X2 as it is connected to a DVI capable monitor. This machine runs Ubuntu and used to be home to the Galaxy 8600GT. The old 8600GT has been moved to the Windoze Celeron as it has a VGA plug.

The new card has been running overnight and has completed a couple of SETI work units as well as a GPUGrid work unit. I won't get too excited yet as the Galaxy only started presenting problems after a couple of weeks.

9800 continued...

Got an email from the computer store to say that they can't find anything wrong with the Galaxy card, however they will swap it for a Leadtek equivalent. I'm happy with that as the 9400GT Leadtek hasn't given a moments trouble since I put it in the same machine that the Galaxy 9800GT was in.
Hopefully I will have the replacement by the end of the week.

2.5 Million!

Well, reached the 2.5M credit mark this morning. My average credit has dropped to below 9k/day ... so the next mile-stone will be some way off. SETI@Home is short on work so the two slower CUDA cards aren't even earning their keep.
The 9800GT is on its way back to the supplier this morning via the post office. I'm only about 50% sure I will see it again...

End of the Galaxy 9800!

Well, if you look back over the last 4 months that I have had the Galaxy 9800GT card, you have to admit that I have tried almost everything possible to get some credit out of it. It has got to the stage where every SETI and Einstein work-unit stalls after a few minutes and despite running a CPU Benchmark every hour, most times the cant be "unstuck". The GPUGrid work-units error out after a minute or two without fail.
I have removed the 9800GT from service and will attempt to send it back to the on-line store that supplied it. I'm disappointed to say the least as it cost me over $200 at time of purchase and gave problems the whole time. I would strongly recommend that anyone running BOINC avoids this card at all costs!

Might be on to something ...

Having watched carefully, it does appear that my Nvidia 9800 based cruncher works fine for a week or so after the drivers and application exe's have been installed and then slowly but surely the work-units start to hang until no work is done at all.

I'm a bit stumped by this ... maybe files are getting corrupted on the hard drive due to excessive use??

BOINC installation gets "tired"

I upgraded to the newest Nvidia driver as detailed in the last post and GPUGrid ran for around 3 days. Then every single work-unit started to fail ... so, I set it loose on SETI and for almost a week it did really well and then a huge batch of 500 work-units was trashed. Restarted the machine after the weekend and almost every work-unit is failing. I have just gone back to the 182.50 driver as the general consensus is that this is the most reliable one.
So, any ideas? It really appears as if the installation "gets tired" or is corrupted?? Just to add another couple of variables, I have installed the Einstein Beta CUDA client ;-)
We will see what this week brings...

Updated Nvidia driver for Windows.

My ill-fated 9800GT has been limping along in a Windoze 2000 Celeron and has been experiencing a problem where work units hang after a period of time (usually 20 minutes to an hour). They continue after being suspended and resumed.
I upgraded the Nvidia driver to the latest 190.38WHQL version from the 182.50WHQL version.
So far things are looking good, it has been running a GPUGrid "GIANNI" for about 5 hours without the need for intervention.

More master browser ...

In trying to sort out the Samba server issues, I was using a Ubuntu desktop to try and figure out what was happening on the network. Things were being complicated by what I later discovered to be a bug in Ubuntu's Gnome version. The Nautilus browser does not know what to do with "smb:///" and as a result makes it look as if there is no browser on the network when you click on "Windows Network"
The URL "smb://workgroup/" works perfectly...

On the BOINC front, I have joined AQUA@Home as I am interested in the multi-threading idea in BOINC to use multiple processor cores for a single work-unit. An added bonus is their CUDA support for Linux. Just my luck, they are now out of work!

The mystery of Samba and the Master Browser

In a Windoze workgroup you need one of the machines to be a "master browser" that keeps a record of the machines on the network and allows each machine to access this information in order to browse the "Network Neighborhood". The way this is done is to "rank" each machine according to its operating system (i.e. Windows XP beats Windows 98 and Windows 2003 Server beats Windows XP etc). If there is no current browser, an "election" is held and the machine with the highest ranking wins. The "Browser" service in Windoze handles this.
I have a Samba file server running on Ubuntu and it is set up to be the master browser as it has the capacity to handle more tasks and is almost always available. The problem is that in a mixed OS workgroup (Ubuntu/Win XP/Win 2000) this is not working and when any of the users attempts to browse the "Neighborhood", they receive a message that says they do not have rights ... Annoying! Running Wireshark revealed that the "__MSBROWSE__<01>" key is not set on the Samba machine or any of the others.

I have spent a lot of time Google-ing the issue and it seems to be a common problem but not many good ideas are around for resolving it! The usual "make sure your Samba machine is set up as domain/local/preferred master in smb.conf" and "make sure your OS level is high to win an election" don't seem to sort it out!

I have tried various options in Samba and the following seems to be a winning combination:

1. Allow guest access on at least one share.
2. A OS level of 65 seems ok but wont beat a MS Server Domain controller if present which is good (DC must be master browser)
3. Make sure "global" is chosen as the default service.

I'm not sure exactly which one is the magic setting but I am able to browse the network again...

2 Million Credits!

Finally reached the 2,000,000 cobblestone mark on BOINC today. Been a SETI@Home member since 11-Jun-99 and Moore's Law has definitely been apparent in my credit history.

9800GT drags on...

I sent an email to the vendor that I bought this card from and he agreed that I could return it under warranty. I was planning to remove the 8600GT from my home computer and substitute it for the 9800GT while I waited for a replacement. In the two days it took to run down the work units on my home machine, I did some reading on a couple of the project forums and took note of comments by MarkJ and others that the NVidia Windows driver that performed best was the older 182.50 version. As I am approximately 4 days away from hitting my 2 million credit mark, I decided that it may be worth trying this before ripping the 9800GT out.
So far it has properly installed on the old Win2K machine (which none of the later versions do, they all miss out on dll's required for the control panel etc) and has returned 1 GPUGrid and about 6 SETI work units without a single failure.

As they say "If it seems too good to be true, it probably isn't!" ... but will wait and see how it goes. The next step is to try the recently found Galaxy software to under-clock the card a bit.

And still more 9800GT ...

As a last ditch attempt to get some credit out of my much disliked Galaxy 9800GT card, I have taken it out of the AM2 machine running 64-bit Linux and put it into a 775 machine with a single core 3.06GHz Celeron running 32-bit Windoze. This machine previously was home to a heat-pipe cooled 9400GT running on the NVidia 185.85 driver. On start-up, the driver defaulted to the VGA driver but after the hard-drive churning away for a minute or so, a restart occurred and the screen resolution came right and off it went straight into a GPUGrid work unit.
I left it running overnight but cant see any returned work units this morning. Hopefully it just hasn't done an update yet ... I'll see when I get to the office!

Update: Nope, it's trashing work units even quicker than normal ...GRRRR!

Update 2: Upgraded the NVidia drivers from 185.85 to 186.18 and BOINC from 6.6.31 to 6.6.36. So far have completed 3 SETI work units without a failure. Holding thumbs again!

Update 3: No, still trashing work units in SETI and GPUGrid ...

Admitting defeat...

Well, the 9800GT just goes from bad to worse! It hung for 19 hours this weekend on a SETI work-unit at 0% complete. Re-installing the drivers and upgrading the kernel had no effect. I am going to give up on GPUGrid as I haven't got any credit from this GPU for a while now. I am just going to run SETI when I am actually in the office as about 60% of those actually complete. To do my bit for the planet, I'll turn this machine off at night.
Guess I am out of the credit race :-(

Still having problems with QMC work-units but so are most other Linux users! I'm happy to see that their new Orca application is available as a 64-bit application ...

The 9800GT saga continues...

The number of work-units failing on my Nvidia 9800GT card have been steadily increasing to a point earlier this week where it trashed about 80 SETI work units and 8 GPUGrid units in a row and won't complete anything new. Yesterday afternoon I noticed that my display was missing some text and all the screen effects (fading windows etc) were taking ages. I turned off all the effects and the problem went away, it then occured to me that the Nvidia drivers would be used for any animation effects etc. I have now re-installed the 180.44 driver available for Ubuntu and I have already seen a couple of SETI work-units being returned as complete.

I'm hoping this improves my situation and allows me to get some credit for a change. Here's holding thumbs!!

Future Plans ... "Junk Mountain"

I was looking for a serial port back-plane in my store-room during the week and it struck me that there are at least 3 PCs lying in there. Nothing really wrong with them, just old single core machines (Celerons/Athlons) that have been "decommissioned". There is a link on the BOINC website to a Linux distro (based on Ubuntu) from Dotsch_UX that allows one to set up a very basic OS with Boinc-client that is diskless, on a USB stick or else on a hard drive.
My plan is to investigate the most power efficient way of putting three mother boards in close proximity (no cases etc) with a single drive on a "server" to run the whole show. Running Milkyway's optimized science app, this should churn out a useful bit of credit for a relatively low KW/hr rating.

On the 9800GT front, the GPU seems to be behaving as long as I reboot the PC on a regular basis. Every second day seems to keep it happy.

I'm still getting a lot of client errors on QMC work units. Its not just from one machine, they are coming off 64-bit Linux, 32-bit Linux and 32-bit Windows. There is still nothing much being said on the QMC forums ... it can't just be me this time??

9800GT strikes again!

It has been a long weekend in NSW (Queens Birthday) and my Ubuntu 9.04 AMD X2 with the 9800GT has been on since Friday morning. Sometime on Sunday the GPU card went mindless and returned 240 SETI work-units with 60 something seconds of work and "The number of results detected exceeds the storage space allocated". It also returned 11 GPUGrid work units with a multitude of "unspecified launch failure" errors.
I thought that this machine is no longer on a UPS due to the GPU's power requirements and maybe a power glitch caused the problem. I have seen a fair number of entries on the UPS logs over weekends. There is, however, a Windows machine with a 9400GT on the same supply and it seems to be unaffected, so I think that ruins my theory.
The 240 trashed units actually moved me to go into the office on a public holiday and re-start the machine. There have been no work units returned to either SETI or GPUGrid since. I'm hoping this just means that there hasn't been a need to report completed units yet ... either that or some serious PC surgery awaits me tomorrow morning!
I'm running out of ideas to get some reliability out of this machine! It isnt overclocked, GPU seems to run around 70'C and the CPU work units seem to have no problems (which in my mind rules out HDD/RAM/Motherboard issues)

QMC work units failing on Linux?

My new server is humming along beautifully, apart from a error in a webmin (remote web based admin) configuration module for clamav (anti-virus), all has been well this last week. I'm happy with the decision to go for Ubuntu instead of MS Small Business Server and have actually considered offering to build servers for Sydney clients as I believe there is a market out there.
The one issue that hasn't been resolved is that QMC work units crash most the time on this machine. I have installed the libc++ ver 5 library as well now and the problem still exists. The weird aspect is that this machine derives most its credit from Einstein which is also a 32-bit application (and that runs fine!). There are no QMC failures on my Windows machines. The work units that fail on the Linux machines also fail on other computers that receive them (both Windows and Linux) with "exit code 24 (0x18)" but are eventually completed after a couple of attempts. There does not seem to be much on the QMC forums that suggests other people are noticing this.
The one aspect I have noticed is that the problem is worse in the machines that are throttled via the BOINC setting. Unfortunately this is not the cause as the problem also occurs in an AMD X2 that runs at 100%. My current course of action is to install all the GLUT (OpenGL Utility Toolkit) libraries and see if that makes a difference as the QMC application does call "freeglut" according to the logs.

Update on reorganisation

Well, the quad server is up and running happily on Ubuntu server and all the features like Samba "oplocks" have been sorted out allowing clients access to the databases. On the BOINC side, this machine is crunching Milkyway, Einstein and Rosetta with 85% processor usage.

The machine that has the suspect 9800GT was tagged to become a Windows 7 machine however after three attempts at installing 7 (all ending in blue screens), I went for Ubuntu 9.04 desktop in 64-bit flavor. At the moment I have 3 consecutive GPUGrid work units that have completed successfully. I'm hoping that there was either a problem with the machines OS or else it was just a symptom of the hassles with GPUGrids latest work units (still doesn't explain the crashed SETI Cuda units...)

The latest upgrade is a 3.06GHz Celeron that has now been moved into out workshop area to run some equipment. This is replacing a rather temperamental 2.4GHz Celeron. As I was a bit concerned about heat in the case and didn't want to go for case fans ets, I opted for a heat-pipe cooled 9400GT CUDA card. I didn't read the specs very carefully and was a bit sad to note that it only has 16 stream processors opposed to the 112 on the 9800GT. I installed the latest driver off the Nvidia website (185.85) and everything seems to be up and running with little pain. I am a bit sad to see that the latest BOINC software doesn't have the option to run as a service (or am I missing something again??). While the 9400GT won't make a huge contribution to the credit total, it will churn out some SETI work units and hopefully some Milkyway work units (when CUDA work is available) in order to justify the power the PC consumes during an average work day. The other plus side is that it is the first time I will have the oppertunity to play around with CUDA/Boinc on a Windows machine.

It's official ... I'm cursed!

The office Quad is up and running (more on that later) on Ubuntu 9.04 Server 64-bit and one of my first steps was to install the Boinc client and get it crunching, I chose QMC as they have a steady flow of work units and there is no optimised applications to try and install with no GUI. One of the things I learnt a while back is the QMC have 32-bit applications, so to get them to run on Ubuntu 64-bit, you need to install the ia32-lib package.
Imagine my surprise when I looked on the QMC website this morning and all but one work unit had crashed. This is a brand new install on "server hardware" with good memory... this along with the hassles with the GeForce card on the AMD X2 is making me think that someone somewhere is sticking pins in a doll! My solution was to detach from QMC and attach to Rosetta and Einstein instead, if these also crash, I have problems!

A bit more about the new local server for the office, it is a cool running Q8200 with 4Gigs Corsair RAM and 1 Terra byte Hitachi SATA2 7200rpm drives in Software RAID 1. It makes a MS Small Business Server look a bit unnecessary and also a bit pointless ;-)

• Ubuntu "Jaunty" 64-bit server - Operating System
• mdadm - Software RAID 1
• webmin - Web based config GUI
• samba - File sharing
• squid - Caching proxy
• havp + clamav - Anti-virus proxy for http traffic
• procmail + spamassassin - Anti-spam mail handler
• dovecot - Imap & POP mail server
• apache2 - Webserver
• mysql5 - Database server
• bacula - Backup daemon

The set-up has been time consuming and I have battled to get the authentication working for file sharing for Windows clients but I am not sure how much of that has been caused by saved passwords etc on the XP machines and how much has been caused by me mis-configuring samba. Im also getting some 417 errors when client machines try and talk to BOINC servers and I gather it has to do with squid only talking HTTP 1.0 and not HTTP 1.1. The strange thing is that it works fine on the “soon to be retired” proxy (also squid on 64-bit Ubuntu). I can thankfully just turn off the proxy connection in the BOINC managers … but it would be nice to have again!

I have downloaded optimised clients for Seti and Milkyway, so as soon as the machine has been installed and is stable, I'll see what credit I can squeeze out of it!

The end of the 9800GT?

I have installed a later driver, re-installed the boinc-client, detached and re-attached to the GPUGrid project and the majority of the work-units on the Linux machine are still failing. I am left with the conclusion that it must be a hardware problem as the card earned 160,065 credits before developing this fault. I assume that 24/7 operation is more than the Galaxy card can cope with in terms of heat dissipation etc ...
That was an expensive bit of hardware for a month's credit! To add insult to injury the vendor concerned has dropped the price of this card over $50 in a month, could be that they know there is a problem? I can't get the nvclock package to work as I was hoping to under-clock the card to see if that helps. I see other users are also experiencing failures on GPUGrid and SETI ... so will leave it for a couple of days and see if it comes right...

On the positive side the 1TB drives arrived for the RAID1 setup in the new Quad. Along with this is a GeForce 9400GT with a heat pipe and no fan. Hopefully this will last longer than the fan-cooled 9800GT.

Lost in the post

I received the case, motherboard etc for the new machine however the second carton of the consignment didn't make it (thanks, Auspost!). So, no drives or GPU. Hopefully they turn up at some point...

The 9800GT in one of the 64-bit Ubuntu machines is getting rather unreliable and is failing work units in both Seti and GPUGrid. I'm not sure exactly what the problem is as the fan on the GPU is still running and the errors seem pretty random. I am considering turning this machine into a Windows 7 machine so that I can run some of the Nvidia utilities and maybe reduce the clock speed to get some reliability. Its not overclocked at the moment but maybe running at 100% 24/7 is just too much.

Our ISP at the office has changed the way in which they scan email for spam and as a result we are getting massive amounts of it at the moment. I have installed a program from the Ubuntu repository called "POPFile" that catagorizes mail according to patterns it has learnt from mail manually checked. It seems pretty effecient so far!

New Quad on the way

Well, so much for no hardware budget :-) The company has just funded a new local server that will take the form of an Intel Quad with a healthy dash of RAM and 2T of disk space that is destined to be run in a RAID 1 configuration. A GeForce 9400GT with heat pipes is included in the order as I believe this card will churn through a decent number of SETI work-units without adding too much heat to the case. The ETA on this machine is the middle of next week. The IBM x-Series server will be retired due to a lack of space and the current local server will replace an old Celeron in the workshop area. The OS is going to be Ubuntu 64-bit (big surprise!) but I’m unsure of which version to choose. At this stage I am leaning towards the Long Term Service (LTS) release of 8.04 Hardy Heron as this is supported until April 2011 but am a bit concerned as I know the software RAID has been simplified in the later releases. I will have to do a bit of reading!

Windows 7 on VirtualBox OSE on Ubuntu Jaunty

While this doesn't have anything to do with BOINC, I had to comment on a bit of potential insanity I commited yesterday. I have been wanting to have a look at Windows 7 as we still use XP Pro at the office but dont have a spare machine with a DVD-Rom to install the Release Candidate on. I wasn't really expecting this to work but decided to try and install Win7 on my IBM x-Series using the VirtualBox OSE package from the Ubuntu Synaptic Package Manager. What I really liked about this program is that you dont need to burn a DVD, you can just use the iso file off the desktop.

My hats' off to the creators of VirtualBox, it all worked first time and Windows 7 installed and runs fine. With a bit of playing around with the network settings, the virtual machine is visible on the general network and the desktop can be accessed by Microsoft's Remote Desktop connection. I have also installed a trial version of the latest MS Office and a couple of people are giving it a test-drive.


Im now just resisting the urge to install BOINC on the virtual machine ...

Upgrade to Ubuntu 9.04 64-bit

Today was the day and I upgraded my home PC (the Athlon X2 4200+ with 8800GT) to the latest version of Ubuntu. As I have messed around and installed a lot of junk, I decided to do a clean install instead of allowing it to do its own upgrade. I had a niggling feeling that getting BOINC and CUDA to work again would be a little bit of a challenge... and was right.
When the new OS booted for the first time, it decided on a generic display driver and as a result decided that 800x600 was the best resolution. Unlike the previous releases there was no option of just turning on the restricted drivers to get everything to work. I had to go and find all the NVidia drivers in Synaptic and do a few restarts to get X to work. Finally got the 180.44 driver on BOINC 6.4.5 (there was no GetDeb option for 9.04, so had to use the old 8.10 version which seems to work OK)
So, after about 3 hours we are up and running again ... except SETI appears to be down :-(

Best day ever!

I got the grand total of 14 987 credits yesterday (22nd April) which is the higest I have ever recorded. Of that 12 428 came from the Geforce 9800 GT (83% of total). Lets see a PS3 beat that ;-)

GPU Credit

The PC with the Nvidia 9800 GT has been running for a few days now and the increase in credit is amazing. A single GPU worth $175 is out-performing a brand new computer that I would have paid over $1000 dollars for.




This graph from BOINCStats gives an indication on how the credit/day has improved from the standard AMD Athlon X2 4000+. Mention must be made that I am only only running BOINC projects on one CPU due to the high usage problem with the Linux CUDA client. Things will get even better once this has been resolved.

Teething problems

The Geforce 9800GT has been running for 24-hours and two issues seem to have arisen. The first was a "stalled" SETI work-unit, while most of them are taking around 18 minutes, one took over 8 hours with no evidence of a problem. The CPU time taken was just over the 18 minute mark but when looking at the start and finish times, the wall time was over 8 hours. It wasn't even a VLAR work-unit. I have no idea what the problem is/was but will watch and see if it happens again.

The other issue is something I should have thought about before installing the GPU ... the UPS!! I have the two "local servers" plugged into a Powerware 3105 UPS. The idea at the time was to even out any power spikes and have around 60 seconds to power down the two machines in the event of a power failure. With the addition of the 105W GPU, I am now over the UPS's limit when the SATA drives are running in the file server.
As a short term solution, I have taken the hub and router off the UPS to give me a little more capacity but am going to have to think of a more elegant solution to this problem.

19 minute work-unit

For once I was proven wrong, with the knowledge gained when installing the 8600GT card under 64-bit Ubuntu, the 9800GT wasn't a challenge. I installed the new 460W power-supply and was pleased to see that it had a 6-pin power connector that would plug directly into the 9800GT without an adapter.

I was initially confused by the air-flow in the Nvidia card, but discovered that the fan pulls air into the card housing and it vents near the blanking plates on the back of the case. As a result, I pulled the two neighboring blanking plates out (not shown in picture) and it seems as if a fair amount of heat is now expelled out of the case despite the power supply fan trying to pull it in. There is no case fan in this PC and I will monitor temperatures carefully for a couple of days. At the moment, the GPU's core temperature seems to sit around 62'C with an ambient temperature of 50'C.

Once again I have run into the anomaly where the Boinc client wont recognize the CUDA card when it starts. I have to close the client and restart it with a "sudo /etc/init.d/boinc-client start" command (the same as the 8600GT installation). This is a minor irritation as the 9800GT machine will run for the entire work week before being turned off. I have a suspicion that this may have something to do with the user "boinc" being added to the "video" group but not having sufficient rights.

Anyway, put the case back together after checking to see if it all started and booted OK, and am happy to see SETI work-units being dispatched in 19 minutes on average. This is under half the time taken by my 8600GT with the same OS.

On a different topic, Ubuntu officially release their 9.04 version in 13 days. I am running the beta 32-bit version on the IBM server and it appears to have some nice features and refinements over the current 8.10 version. I will re-do the OS on my home computer (the AMD X2 with the 8600GT) but will leave the 9800GT's OS on 8.04 as it is a "Long term support (LTS)" version and it is very rare that I even log into this machine. The proxy and backup functions can all be controlled via an Apache webserver interface.

Galaxy GeForce 9800GT

When the whole "SETI on CUDA" idea first came about I bought a second-hand 8600GT just to see how the concept worked. A couple of months down the line, I have come to realize that ordinary CPUs aren't going to keep pace with CUDA (and other future GPU standards). So when the Australian government recently announced their financial stimulus package, I decided that this was a loop-hole I could use to upgrade the "farm" without a budget.


A brand new Galaxy branded GeForce 9800GT is the result! The specs are as follows:


Stream Processors: 112
Memory: 512MB
Texture Fill Rate (billions/sec): 33.6
Memory Bandwidth: (GB/sec): 57.6

As the majority of my PCs still use AGP slots, the options of where to install the new card are limited. As a result, I also bought a 460W power-supply to upgrade the 64-bit AMD based Linux machine at the office. This system runs the Squid proxy as well as an anti-virus proxy and does the backups of the file server each night. For this reason, it runs 24/7 and has a fast ADSL2+ internet connection.

The down side so far is that the Nvidia drivers for this OS only go up to version 167, so after some time on Google, I found a site that allowed me to download 180.44 drives for amd64 architecture.

As I cant take this machine down during office hours as I risk the ire of the Facebook junkies in the building, I am hoping to spend an hour or two at work over the Easter weekend and see if I can get it up and running... but have learnt from experience that these things are never as easy as they seem!

Ubuntu 8.10 to 9.04 beta ... Beware!

I have, in the past, just gone for the "upgrade to new version" in Ubuntu's Update Manager and have been lucky. Version 8.04 to Version 8.10 was an example of this. I wasnt so lucky this time around as my IBM x-Series got as far as "Starting HAL" and then died a horrible death.
I downloaded the ISO file and re-installed from CDRom but in the process lost a notable amount of half processed work-units (mainly SETI Astropulse).
All is up and running again and the new version does look good, but this is a lesson to back up regardless of how well things have worked in the past

April Fool

I ran the cache of SETI work-units down and then detached and re-attached to the project with the view to download the 64-bit Linux CUDA client. The reason for that is that the optimized client from Crunch3r uses 100% of a CPU core while it runs. I was hoping that SETI's own client would use less and enable me to get some productive crunching out of the "wasted" core.

What I didn't realize is that SETI still haven't released a client ... well, I was unable to find one or get it to download...

So back to to Crunch3r's client and now just a bit of a delay to get full-time work for it. SETI have had a hardware problem this week and as a result have run short of work to distribute.

Shutdown for Earth Hour

To mark this years Earth Hour, all the machines at the office (including the servers) have been shut down for the weekend. The only machine crunching at the moment will be shut down for the hour at 20:30 tonight.
Running older hardware does often worry me in terms of the credit/watt rating I achieve, but I guess it is still more productive than all that hardware out there that sits idle while chewing countless kilowatts.

BOINC client update through Synaptic on Ubuntu

The version of BOINC that is included in the Ubuntu default repositories is rather outdated. I used the deb file from www.getdeb.net to install a client (6.4.5) that was recent enough to allow me to run a CUDA card on my 64-bit Ubuntu installation. I am the kind of person who will never remember to check for updates, so I have added getdeb.net to the repositories so that Update Manager and Synaptic can see updated packages.
This is achieved by adding "deb http://ubuntu.org.ua/ getdeb/" to the Third Party sources in Software Sources (found in System > Administration > Software Sources). This will also update a number of other packages on your computer, so make sure you are happy with the security and version numbers before you go ahead.

SETI for CUDA on Linux vs Windows

Well, the question posed last week regarding the big difference in performance between my CUDA card under 64-bit Ubuntu and cards under Windows has been answered.
I finally spotted an instance where two 8600GT cards have received the same work unit. The machine (AMD 64 X2 6000+) running 64-bit Windows finished the work unit in 156.45 seconds while my machine (AMD 64 X2 4200+) running 64-bit Ubuntu finished in 2968.83 seconds. That is a difference of 94.37%!
Looking at the CPU bench-marks, the Ubuntu machine is actually quicker so that shouldn't influence the result. The only notable issue is that the Windows machine has double the memory, both RAM AND GPU memory.
It will be interesting to see if this major difference still exists when SETI release their own Linux client...


After some time searching forums for ideas, the following two concepts were suggested by a user on the lunatics.kwsn.net forums:

1. 256M of GPU memory is right on the border and this may cause the work unit to fall back to CPU computation. I'm not convinced about this as I have watched the GPU temperature and there is no evidence that it stops crunching.
2. The linux CUDA application does use 100% of a core and I was just thinking that the "0.04 CPUs" line was a complete joke. The time reported could well reflect the "wall time" for the work unit, Windows on the other hand does use a small percentage of a CPU core (I believe..?), so the time reported to complete a work unit isn't necessarily anywhere close to "wall time". If this is the case, a Windows machine essentially has another core compared to the Linux machine and will have a higher credit average ...Grrrr!
   

Comparative Performance of CUDA Cards

The question in my mind at the moment is “which Nvidia card is the best value for money for SETI?”. I have gone through the results from the CUDA work-units and pulled out the ones where the credit granted is the credit claimed (i.e. work units where both computers used GPU).
I have used my machine as a reference with its 8600GT card with 256M RAM. The potential problem is that I am running the Crunch3r CUDA client on 64-bit Linux and every other result came from Windows machines. This may account for he big difference between my GPU and the next slowest on this list which is a 8800GT...?

In this table, the speed of the GPU cards are the number of identical work units they could process in the time taken for the 8600GT to do a single work unit.

8600GT(256) Linux : 1
8800GT (512) : 21.3
9600GT (512) : 25.3
9800GTX (512) : 23.1
GTX260 (896) : 17.0
GTX285 (1024) : 24.2
GTX295 (896) : 23.2

Out of that lot, the 9600GT seems to be the first choice! I will spend a bit more time on this and post a more comprehensive list as I get more data.

Upload problem with SETI

There appears to be a problem with SETI@home uploads, at first I blamed my Unwired internet connection but quickly came to the conclusion that it was on SETI’s side. Their website forum hints at bandwidth problems resulting from Astropulse activity. Needless to say, I have dropped a few places in the rankings over the weekend.

The Unwired connection at home has not been without its share of problems in the past. When I moved into my apartment, I signed a 6 month lease and it didn’t seem like a good idea to take a 12 month DSL contract. At that stage there wasn’t much available in the GPRS/3G market that wasn’t expensive and didn’t require a 24 month commitment. I decided that based on the fact that Unwired supplied an Ethernet modem (much easier to get working on Linux as there are no drivers), I would give them a try despite reading some pretty bad reviews.
Unwired uses the 3.5GHz range and when I first tried to get a connection, there was massive interference in the area. This would disappear around 10pm and then start again at 6am … not useful when wanting to browse and check email in the evenings!
As I had paid for the modem, I didn’t want to admit defeat so I moved it around my apartment for a frustrating couple of hours until it landed in the current location (taped to the inside of the lounge window). I tried removing the fold-up antenna but that didn’t seem to make a difference although it did stop the device roaming from the side to the main antenna and back every couple of minutes. The heavy baking tray in the picture is to try and screen the "wabbit" from interference that seems to emanate from directly opposite to the Unwired tower. There is a D-link wireless router on the floor that is connected to the Unwired modem that eliminated the hassle of running Cat5 around the lounge.

Well, a couple of months down the track and I get peaks of around 90Kb/s which is adequate for what I do, although the service is still far too unstable for Skype with drop-outs every couple of minutes. I noticed that they have upped the speed on the prepaid service to 1024/256 ... thats an improvement!
I may try a 3G service at some stage as they are now cheaper but the idea of trying to get a USB modem working on Ubuntu doesn’t appeal!

Optimized SETI App and Norton AntiVirus cont.

After some time with Google, I have found a couple of references to other people having similar problems with false positives from Norton Antivirus 2009 and Norton Internet Security 2009 with the AK-v8 application. Based on my experience, it also affects the 2008 version of Norton.

Optimized SETI App and Norton AntiVirus

After installing Crunch3r's 64-bit Linux CUDA SETI client, I decided that I would look at installing the AK ver 8 optimized clients on a few of my other machines. It does make a notable difference, on the IBM X-Series, the average run-time was around 23 000 seconds and with the SSE2 optimized version, this came down to around 14 000 seconds. I have since installed the SSE3 version and expect a further improvement.

The interesting bit was with my laptop, its a Core Duo that runs 32-bit Windows XP. I checked using a program called "CPU-Z" and it supports SSE3. I downloaded the zip file and installed the appropriate files. When I download anything at the office, the proxy checks traffic using Clam antivirus, the server share it was downloaded to runs a McAfee product and my laptop runs Norton AntiVirus 2008. None of them complained about the downloaded zip file or its contents.

I restarted the Boinc client and I get a collection of windows popping up to tell me that Norton's "SONAR" has found a file infected with "Downloader" and has deleted it. It turns out that it doesn’t like the ak_v8_win_sse3.exe file and has removed it. I also get a warning that an Excel document has been downloaded to one of my network drives.
I copied the "offending" file onto three of our network shares and none of the resident antivirus programs found a problem, so I then upload it to the Kaspersky online service and once again it is clean. There is also no sign or record of the Excel document that was supposedly downloaded.
My question now is: "Does the optimized client have some malicious code in it or has Norton given a false positive?".
I have detached from SETI completely on the laptop as a precaution and to delete all traces of the download that where in the project folder but am now a little nervous about the other machines that have optimized clients installed!
Anyone else seen something similar??

SETI, CUDA and 64-bit Ubuntu

Given up waiting for SETI@Home to release their CUDA client for 64-bit Ubuntu, so downloaded Crunch3r's version. The directions in the "Readme" file are great and its a straight-forward install.
I'm impressed so far, its been running 20 minutes and its almost 20% of the way through a work-unit that would take around 9 hours if crunched on the AMD X2 4200's CPU


Ok, result is 58.73 credits in 2966 seconds!! I think that is pretty good for a second-hand 8600GT with 256Megs RAM.

First GPU Credit

My 8600GT finally yielded some credit for GPUGrid. The total processor time was just over 6 hours but wall time was around 32 hours for the workunit. I'm not sure what is causing the big discrepancy, there was no improvement when I suspended all the other CPU tasks. The credit of 3718 does make the whole thing worthwhile as that is notably more than the CPU (AMD X2 4200+) managed in the same period.

Quintillion

Looking at the "Certificate of Computation" for all projects on the GPUGrid website, I have done 1.06 quintillion floating point operations.
Google’ling and Wikipedia'ing gives a general consensus that a "quintillion" is 10 to the power 18, so I have done 1 060 000 000 000 000 000 floating point operations so far!

Still more cool overcast weather, so my "farm" is running at full steam again today! I bought a 8cm case fan for my home PC (the one with the CUDA card), so hopefully it will be happier during the next bout of hot weather.

IBM xSeries 206

Monday morning and a pleasant 21ºC here in Sydney at the moment, so everything is up and running, including my home PC that is coping fine without case fans due to the side panels having been removed from its case.
The IBM xSeries 206 seems to be behaving itself and the computation errors that occurred on SETI and QMC don’t seem to be a long term problem. The history of this machine is interesting; I was walking past the trash-can area of our office building and noticed that one of the other tenants had thrown out an x-Series case. I slam on brakes and have a look to see if it still has a power supply … score! It does … hang on, it also has a motherboard 2 SATA drives, a CPU and 4 DIMMS. Couldn’t resist and dragged it off to my car.
Somewhat surprised later when the orphan does a POST and starts to boot SBS2003! That’s where the fun ended as a serious blue-screen arrives around login time. After a couple of minutes of playing around with the RAID utility in BIOS, I find a badly injured RAID 1 setup and despite a couple of hours spent rebuilding the array didn't fix SBS2003 . After some contemplation, I decide that Ubuntu RAID 1 looks a little bit tricky (for me at least) so I pull out one of the drives, killed the RAID and installed 32-bit Ubuntu (the IBM didn’t like the 64 bit version!)
The first couple of work-units returned errors and I thought that there must be a CPU/memory/motherboard problem but can't track anything down. Looking at the SETI website, I see the problem work-units all mention SSE3, which this machine doesn’t support. Its got a single Pentium 4 3.0 HT CPU that only has SSE2, the work units that mention SSE2 in the header appear to work fine. No idea what was going wrong with the QMC units, they just appeared to loose the heartbeat!
Apart from crunching, this machine now runs Apache and is visible to everyone on the other side of our router. It is the ideal machine for this as there is absolutely no user data on the drive. It also allows me to use XMing from my laptop when I need Linux with a chunk of RAM (e.g. to run GIMP)

GPUGrid

We all saw this coming! I couldn't wait for SETI@home to start distributing CUDA work units.
Joined GPUGrid and all seems to be working fine, I get "Running (0.04 CPU's, 1 CUDA). The only problem is that my little 8600GT is a bit slow and the "To Completion" time is ticking up steadily.

I'm a bit worried about temperatures until I get another case fan, so have the side off the case and a desk fan blowing towards it. Getting a comfortable 57 degrees Celsius from nvidia-settings.

GPU0 vs GPU1 and CUDA

Well, a couple of hours after I got BOINC to recognize my CUDA card under Ubuntu, I decided that there was a better way to set up my machine.

The computer in question is a home computer that gets used for email and a bit of web browsing and some really old games (via Wine). Generally the graphics card doesn't do much work. As my CUDA card (Nvidia Geforce 8600GT) only has 256M RAM, I decided that I would use the motherboard's built in Geforce 7050M to run the monitor.

Using "nvidia-settings" from the Ubuntu repository, I set the monitor on the 7050, disabled it on the 8600GT and rebooted. I did set the BIOS default card to the onboard 7050 to be able to see the POST and pre-login screens.

Guess what? No CUDA device found by BOINC on reboot!

After another hour of so, it seems as if BOINC just doesn't see the CUDA card on about 50% of startups regardless of whether it is set as GPU0 or GPU1. I resorted to using "sudo /etc/init.d/boinc-client stop" and "sudo /etc/init.d/boinc-client start" a couple of times before I got the required results. While this is OK in winter when the machine stays on for days at a time, it is a pain in the hot weather as I boot, check my mail etc and then turn it off again.

The three "24/7" office machines are turned off this weekend as the aircon is on a timer and as a result doesn't work on the weekends. As Sydney is having temperatures of 40+ Celsius, it seemed like a good idea to loose a couple of places in exchange for working PC's on Monday morning.

CUDA & Ubuntu

After messing with almost every setting on my home computer, it turns out there is one really really important little line that no-one really mentions:

sudo adduser boinc video

It enables Boinc to see the CUDA graphics card! Without this, it just does not work! It seems that you cant do this from the System > Administration > Users & Groups (at least I couldnt find a way!).

I am using the 180 driver from the Ubuntu repository, Boinc ver 6.4.5 from www.getdeb.com and a Nvidia 8600GT graphics card on 64-bit Ubuntu 8.10.

So ... Boinc now recognises the CUDA device ... just have to wait for Seti to start sending out CUDA workunits for Linux!