WCG and GPUGRID resource sharing...

[josh1980]

How the heck does someone "share" the CPU resources between WCG and GPUGRID? I've found that if I set boinc to use 100% of the processors and 100% CPU time my GPUGRID seems to be horribly unreliable at completing tasks within the time it should take.

Example 1:

My Ubuntu box running the latest updates with a GTX 275, 4GB RAM and core i7 920 overclocked to just over 3 Ghz gets 13k PPD! This box has been running since October and the performance of GPUGRID is abysmal. It took 26 hours to get to 50% on one task. I got a little disturbed by this so I suspended WCG overnight. In the 2 minutes I watched it the time remaining jumped by 30 second increments every 3 seconds. I woke up this morning to find that task completed successfully and another task at 40% and crunching right along. Unsuspending WCG seems to slow down the GPUGRID again.

My task duration correction factor for this machine is 11.058098 despite being a dedicated cruncher.

Example 2:

My SR-2 with 24GB RAM and a GTX 480 is averaging just 26k PPD. The system has completed only 424k points since it came online on 11/24. I've seen people say that they get up to 100k/day. I'm getting a mere 38k/day with an overclock to 800core/1600shader. This is with no idle GPU time and no failed WUs. This is my gaming machine, but I game only 2 or 3 hours a day when I play and I even crunch while I play. While I would expect a decrease in PPD on the days I game, I can guarantee I do not cause a 50%+ decrease in performance. My Task Duration Correction Factor is 1.335309.

I do know that when I changed my % of processors used in boinc to 93.5%(15 of 16 cores) PPD seems to increase a little. When I lowered it to 87.5%(14 of 16 cores) PPD seems to increase a little more. Additionally, if I suspend WCG my GPU usage jumped from 63-65% to 74% instantly per evga precision. As soon as I unsuspend WCG my GPU drops back to about 63-65%. I do have the swan_sync=0 as a system variable.

Odd ball machine (example 3): might actually be working correctly...

My HTPC is a GTX 275(same card the linux machine uses) but is windows 7 x64 with 8GB of RAM. It seems to be the only machine that actually outputs even remotely close to what I would expect (35k PPD) but it has random wus that take MUCH more CPU time than the others. Most WUs are 2900 seconds to 3700 seconds of CPU time, but a few are 11000+ seconds. There's no gap in between. It's almost like the machine gets hung up on some wus and it takes much longer to complete. I never do anything on this machine that would tax the GPU. No gaming of any sort. It literally is a dedicated cruncher except about 2 days a week for 2 to 3 hours when I watch TV. I leave it crunching while I watch videos. My Task Duration Correction Factor is 5.187223.

So far I have disabled the screen savers and set power management to high performance. I also changed GPUGRID to a priority of 200(not sure if this actually does anything-different threads conflict as to the actual effect). Now GPUGRID shows up in boinc as a resource share of 66.67% and WCG as 33.33%. All of these changes have been done over the weekend so I don't have any long term history to show if any of my changes have made a difference.

Is there something I don't know about running WCG and GPUGRID? Am I supposed to increase the priority of the gpugrid threads to be above WCG? If so, how do I do this? Can this be automated? I do know that when I look at my history for GPUGRID over the last 45 days or so that I have been crunching the points fluctuate wildly on a day-to-day basis.

[Snow Crash]

I started replying in the PIE thread but now just looked at your machines on GPUGrid directly. It might sound like I'm repeating some of the settings you already have but I'm just trying to be comprehensive for the next person who might need to know ...

Example 1: The new version of the GPUGrid linux app (6.12) runs differently than it had in the past.
http://www.gpugrid.net/forum_thread....rap=true#19362
Add SWAN_SYNC (yes to a linux box) and run 99% of the processors.
BOINC has no idea what SWAN_SYNC is so has no way to account for it.
You need to tell it to leave one alone so the app can grab it for itself.

Example 2: I can almost guarantee that your SR2 Win7x64 running 260.99 drivers is downclocking while you are not looking. The reason it goes up when you are watching is because it becomes "active" in 3D mode and steps down again as a power savings (go ask nvidia). I used a modded inf for the latest quadro drivers and am no longer having downclocking issues. You can also step back to your 258.xx drivers, like your 920 HTPC is (which is why that machine is returning decent results), and the downclocking should stop.
Again .. add SWAN_SYNC (yes to a linux box) and run 99% of the processors.
BOINC has no idea what SWAN_SYNC is so has no way to account for it.
You need to tell it to leave one alone so the app can grab it for itself.

Example 3.
With 258.XX drivers on Win7x64 you do not want to set SWAN_SYNC ... it will actually slow down.

<edit for clarification>
Don't use SWAN_SYNC on any 2XX (or earlier) cards
</edit for clarification>



Overall WinXp runs much better than Vista/ Win7 and is on par with linux.

Steve

[josh1980]

I started replying in the PIE thread but now just looked at your machines on GPUGrid directly. It might sound like I'm repeating some of the settings you already have but I'm just trying to be comprehensive for the next person who might need to know ...

Example 1: The new version of the GPUGrid linux app (6.12) runs differently than it had in the past.
http://www.gpugrid.net/forum_thread....rap=true#19362
Add SWAN_SYNC (yes to a linux box) and run 99% of the processors.
BOINC has no idea what SWAN_SYNC is so has no way to account for it.
You need to tell it to leave one alone so the app can grab it for itself.

Example 2: I can almost guarantee that your SR2 Win7x64 running 260.99 drivers is downclocking while you are not looking. The reason it goes up when you are watching is because it becomes "active" in 3D mode and steps down again as a power savings (go ask nvidia). I used a modded inf for the latest quadro drivers and am no longer having downclocking issues. You can also step back to your 258.xx drivers, like your 920 HTPC is (which is why that machine is returning decent results), and the downclocking should stop.
Again .. add SWAN_SYNC (yes to a linux box) and run 99% of the processors.
BOINC has no idea what SWAN_SYNC is so has no way to account for it.
You need to tell it to leave one alone so the app can grab it for itself.

Example 3.
With 258.XX drivers on Win7x64 you do not want to set SWAN_SYNC ... it will actually slow down.

Overall WinXp runs much better than Vista/ Win7 and is on par with linux.

Steve

I plan to add swan_sync to my linux machine when I get home. I'll have to do a little research to see how to add swan_sync system variable to linux. This machine is a GTX 275, and I thought swan_sync was only for 400 series video cards.

As for the 258.xx drivers with win7x64... that machine is also running a GTX 275. Again, Isn't swan_sync meant only for the 400 series?

It's so confusing staying up to date with everything since it keeps changing. You can read one thread and flip-flop back and forth on almost any topic.

I appreciate you being comprehensive. It seems that being thorough is the best course of action for my issues.

[SMTB1963]

...

SWAN_SYNC (yes to a linux box) and run 99% of the processors.
BOINC has no idea what SWAN_SYNC is so has no way to account for it.
You need to tell it to leave one alone so the app can grab it for itself.

How about dual GPU setups? Is reserving 1 processor enough for 2 concurrent GPUGRID tasks, or does each GPU need its own processor? I have a 2 x GTX275 + i7 920 + Win7x64 machine, and a 2 x GTX460 + 1055t + XP64 machine.

... Isn't swan_sync meant only for the 400 series? ...

Yeah, I'm pretty sure SWAN_SYNC is for GTX400's (and 500s?) only.

[josh1980]

Alright....

On my linux box I have swan_sync=0, use at most 99% of the processors and 100% CPU time.

On my SR-2, I have swan_sync=0, use at most 99% of the processors and 100% CPU time. I have set up a log to monitor GPU speeds, so far no downclocking.

We'll see how this goes

[josh1980]

Well, linux machine is still not working right. After 3 hours 36 mins it has completed just 12% of the WU. It had something like 25 hours remaining. As soon as I suspended WCG it started counting down the time remaining in 20 second increments. So somehow it's not fixed despite all the stuff I've done. Is there a way to set up an automatic task to increase the priority on all threads that meet certain names every 20 minutes or so. I believe this would be a cronjob(?) for linux.

[Snow Crash]

@SMTB - I edited my first post ... added the no SWAN_SYNC on 2XX cards ... ever.
@josh - I don't even run a linux box so I'm just parroting back what the head honcho at GPUGrid said (GDF really is the head scientist, not just a volunteer)
I have seen someone say that adjusting the 'nice' value works just as well/ better than SWAN_SYNC ... but I don' know if they had an automated way to do that.

[WhiteFireDragon]

josh, i had a similar question back then when i first started. no time to type a long post this moment, but maybe you can refer to this thread. snowcrash also did a great job answering it:

http://www.xtremesystems.org/forums/...d.php?t=251476

[werdwerdus]

for dual gpu setups swan_sync will use 100% core/thread per gpu.

[SMTB1963]

I don't mean to hijack josh1980's thread here, but I've been playing around with some of these settings and I'm afraid I'm only growing more confused. Sorry if I'm sounding like I want to be spoon-fed this info, but I've got a few more questions...

I've noticed on my i7 920 + 2x GTX275 + Win7x64 machine that if I set BOINC processor utilization to anything below 100% of processors, GPU utilization drops by about 1/3. For instance, if I have two KASHIF GPUGRID tasks running along with eight WCG tasks (and BOINC processor usage is set to 100%), my GPUs run at 72% or greater. If I set BOINC proc usage to 99%, my GPUs drop to ~49%. If I suspend WCG, GPUs go back up to >=72%. It kinda leads me to believe that the BOINC processor usage setting limits both WCG and GPUGRID CPU access, but my system shows seven threads @ 100% + one thread @ ~50% under the 99% of processors setting with both projects active.

Anyone know what's going on here?

for dual gpu setups swan_sync will use 100% core/thread per gpu.

So in order to maximize GPUGRID performance on my AMD1055t + 2x GTX460 system (while still crunching WCG), I would set SWAN_SYNC = 0 and BOINC processor usage to utilize 4/6th of the 6 cores? Or am I still not getting it?



Hey, if anyone wants to slap me and say "use these settings ya moron! ...", I won't mind in the least.



Any help appreciated!

[Snow Crash]

Just to make sure we are all talking about the same setting in BOINC Manager it is "On multiprocessor systems, use at most 99 % of the processors.

I'm guessing that it only looks like it is the CPU usage percent that effecting your 2x275 GPU crunching and is more likely the GPU driver being *green* and throttling itself down into 2D mode.

I leave Precision open to keep both cores of my old GTX295 in 3D mode. Do not use SWAN_SYNC for 2XX series.

I'm pretty sure SWAN_SYNC is only ever going to use one core in total but I don' have 2x4XX to test that out for you.
Suspend WCG fpr a minute and see what Task Manager has to say about it?

If you are using WCG to manage your profiles they give you an option of a number of cores instead of a percent.

Are you on the XS team? Do you crunch under another name on GPUGrid?

[SMTB1963]

Just to make sure we are all talking about the same setting in BOINC Manager it is "On multiprocessor systems, use at most 99 % of the processors.

I'm guessing that it only looks like it is the CPU usage percent that effecting your 2x275 GPU crunching and is more likely the GPU driver being *green* and throttling itself down into 2D mode.

I leave Precision open to keep both cores of my old GTX295 in 3D mode. Do not use SWAN_SYNC for 2XX series.

I'm pretty sure SWAN_SYNC is only ever going to use one core in total but I don' have 2x4XX to test that out for you.
Suspend WCG fpr a minute and see what Task Manager has to say about it?

If you are using WCG to manage your profiles they give you an option of a number of cores instead of a percent.

hehe, I set to work putting together some screen shots showing BOINC Manager/Task Manager/AIDA64 showing all the relevant settings and metrics...and now I can't replicate the problem! Typical. Like the strange noise coming from under the hood that magically disappears when the car is driven to the mechanic. For the particular GPUGRID tasks I'm running now, there is no dramatic drop in GPU utilization when I set the "...use at most __ % of the processors" to 99%.

Thanks for your reply anyhows.

Are you on the XS team? Do you crunch under another name on GPUGrid?

Yeah, I'm crunching with XS for both GPUGRID/WCG under SMTB1963 - although somewhat intermittently. I hope to be a little more consistent in the near future though. I'll definitely be bringing all I've got for the upcoming GPUGRID Supercomputer Week, which is the main reason for my interest in this thread - i.e., can one run WCG with minimal/no impact on GPUGRID output.

Regards

[josh1980]

Just to make sure we are all talking about the same setting in BOINC Manager it is "On multiprocessor systems, use at most 99 % of the processors.

I'm guessing that it only looks like it is the CPU usage percent that effecting your 2x275 GPU crunching and is more likely the GPU driver being *green* and throttling itself down into 2D mode.

I leave Precision open to keep both cores of my old GTX295 in 3D mode. Do not use SWAN_SYNC for 2XX series.

I'm pretty sure SWAN_SYNC is only ever going to use one core in total but I don' have 2x4XX to test that out for you.
Suspend WCG fpr a minute and see what Task Manager has to say about it?

If you are using WCG to manage your profiles they give you an option of a number of cores instead of a percent.

Are you on the XS team? Do you crunch under another name on GPUGrid?

Yes, on multiprocessor systems use up to 99% of processors. I have proved that none of my machines are throttling down. I do have EVGA precision set to load on startup for my windows machine. It is necessary for me to load it to get my 800Mhz core clock. I am not using WCG to manage my profiles, but I did try that. I did try using WCG to manage my profiles for a little while, but with the number of machines I have it would become inconvenient to manage different settings for each "group" of machines. Not to mention WCG only gives you the ability to create 3 profiles. I am on the XS team and I only crunch under the one profile.

I've since taken the linux machine out of crunching completely. The DFI Lanparty Jr X58-T3H6 has been nothing but problems since the day I bought it. I have called their customer service and gotten worthless responses. The board has been unreliable as could be since the day I started using it. I've changed RAM, CPU, power supply, everything. The computer doesn't always POST, doesn't always shutdown when I try to shutdown from the OS, doesn't always reboot. Today I flashed the BIOS with the latest version hoping to fix the unreliability issues with turning on and off the computer. Now linux won't boot and I don't know where to begin to fix it. I've had it with DFI and the motherboard. If it weren't -25 outside it would be in pieces right now thanks to my baseball bat.

As it stands right now I have no linux cruncher. I have appeared to fix my PPD issues with WCG and GPUGRID running simultaneously by forcing the GPUGRID process to run at higher priority.

My HTPC machine with Windows 7 x64 is currently making about 35kPPD/day crunching both WCG and GPUGRID using priority64 by eFMer set to force all threads with acemd2 to a priority of "normal". Of course, I have also changed my GPU settings to use the computer only when idle for 2 minutes so it doesn't affect my gaming. This has killed my PPD for my SR-2 rig which i'm okay with.

Honestly, I'm running out of patience with this crunching thing. It's not very easy for someone to setup their machine to crunch without knowing a bunch of little secrets like SWAN_SYNC, forcing higher priorities, etc. Don't forget you gotta make sure you have the right version of the drivers installed. One thread on GPUGRID was trying to tell people to use a driver version from 197.xx. Sorry, I'm a little more than pissed that one thread says "always update your drivers" while another says "keep the old drivers". I'm about to say this .

[SMTB1963]

...Honestly, I'm running out of patience with this crunching thing.

As the kids have been out on their own for some time now, I find that I have a huge surplus of patience...maybe I could send you some?

I'm about to say this .

Well, I hope you're not serious about that. These "infernal contraptions" can certainly be frustrating - but I've read more than a few of your posts, and it seems apparent to me that you know what you're doing. You'll figure it out. And I've watched your numbers; you've gotten up there with the other big boys on this team whose efforts and commitment I aspire to. You're making one helluva contribution to some very good causes.



Maybe take a break and do some holiday stuff? A little eggnog and a few laughs with friends never hurts.

Good luck!

[Snow Crash]

Yes, GPUGrid can be frustrating at times but it looks like you have your 2x275 running smoothly now
I don't use eFMer but maybe I'll give that a shot to see if I can increase my PPD a bit more.

While the results from GPUGrid are real, regularly published, presented at biomed and grid computing conferences, the project is still listed as BETA froma a BOINC perspective. They are continually exploring new techniques for computing their app and refining the app itself. They have gone from PS3 to CUDA, they have tried out multi-CPU, and continue to work with each new release from ATI to see if the SDK is robust enough for their app.

OK, I'll play the jerk ... skgiven over at GPUGrid is only a volunteer and while doing the best he can to help people, he post all the time and frequently he posts crap. Without any testing, he makes up results he wants to see and then tries to azz backwards into the logic without presenting any methodology ... more than once myself and others have asked him to stop using this thought process to no avail.

As for drivers ... every now and then a driver release includes an upgrade to the CUDA libraries and GPUGrid is always upgrades their apps to take advantage of something new/ improved in CUDA. This was the case with the 260.99 drivers release. The only real problem with the newest drivers has nothing to do with GPUGrid ... the darn things slip into 2D mode all by themselves. While GPUGrid does test on Linux, they simply don't have enough time/ money to test all the combinations between driver versions/ OS/ card families so we try and help as best we can. They also recently changed how the Linux version of the app runs which is also creates confusion on the boards over there.

[josh1980]

@ Snow Crash

That clears up alot of the "mystery" of GPUGRID. My main reason for ditching Linux and switching to Win XP x64 this morning was because GPUGRID just seemed to "work" on my Win 7 x64 machine using the priority changes. I'm thrilled to say that over the last 12 hours I've had no shifts to 2D mode, the GPU usage is constantly at 93%+, and in the 12 hours that the machine has been crunching I've made 22k points.

Unfortunately, my spare EVGA GTX 260 doesn't seem to crunch without causing errors. I've tried the card in 3 different machines with 3 different drivers using both Windows 7 and XP and it seems to cause a computation error within 45 minutes of every WU I give it. I thought the EVGA OC scanner might be able to shed some light on the cause for errors, but there was error found. I've had no problems with the card before and never overclocked it. I'm at a loss to diagnose the cause since I haven't been able to prove the card is bad, nor have I proved that the issue isn't the card. I only have 1 GTX 260 (the rest are GTX 275s or GTX 480s) so I can't prove that another GTX 260 would perform correctly in my machine. I've tried manually increasing the fan speed to lower temperatures and tried underclocking the GPU but neither one seemed to fix the issue.

Interestingly, I have proved that disabling DEP in Windows 7 x64 Ultimate results in an increase of 5-15% GPU usage. Of course, I would recommend this only for dedicated crunchers. My Windows 7 machine currently sits at 74% GPU usage and is steady. Before disabling DEP I had never seen GPU usage above 60% but it has currently sat at 73% for the last 3 hours. Unfortunately I haven't been able to find a way to exclude all executables in a given folder (for instance C:\BOINC\WWW.GPUGRID.NET\*.exe) but have to disable DEP completely. You can disable DEP for one single executable, but when the next one is downloaded(and has a different name) DEP would be enabled again.

[josh1980]

Well, I found that DEP settings are stored as registry keys. The location is Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Win dows NT\CurrentVersion\AppCompatFlags\Layers\. I created a new string value of E:\BOINC\projects\www.gpugrid.net\*.exe and set the Data field to DisableNXShowUI and rebooted. My GPU temp went up 3 degrees but my GPU usage as reported in EVGA precision stayed the same. In BOINC the GPU time remaining seems to be increasing by 2-3 seconds every second and the WU is 26% complete. I would have expected that at 26% the time remaining would have been fairly accurate by now. In the 12 minutes of uptime with the new DEP setting I've lost 29 minutes for the WU. I also disabled the antivirus from scanning the boinc folder which is recommended but I've never actually implemented.

[Snow Crash]

The old 260s don't work on GPUGrid but the newer ones do.
I'm too lazy to look up the exact shader count for each but I think the old is 296 and the newer 216?

A lot of the GPU usage and overall runtime depends on the WU type. There are some that run fairly fast and hot (KASHIF) and most of the others run slower with lower utilization. Then there are the WU that say *long* in their name which can take a VERY LONG time to complete. Please make sure you are comparing like WU types and even then there is some variation in runtimes.

I really appreciate the dedication and detail you are brining to the table ... looks like I may need to go in for another *testing* phase ... upping priority with eFMer and now DEP reg keys ... nice I typically test with the same exact WU so I can better compare resuts. I do take a hit on points for a week or so but I think it is worth it in the long run.

p.s. WinXP is MUCH more efficient than Vista/ Win7 ... between 20-40% faster runtimes easy

[josh1980]

The old 260s don't work on GPUGrid but the newer ones do.
I'm too lazy to look up the exact shader count for each but I think the old is 296 and the newer 216?

A lot of the GPU usage and overall runtime depends on the WU type. There are some that run fairly fast and hot (KASHIF) and most of the others run slower with lower utilization. Then there are the WU that say *long* in their name which can take a VERY LONG time to complete. Please make sure you are comparing like WU types and even then there is some variation in runtimes.

p.s. WinXP is MUCH more efficient than Vista/ Win7 ... between 20-40% faster runtimes easy

AAh. My GTX 260 is probably 2-3 years old. I'd bet it's the "old" card.

As for different WUs taking different times to complete, I typically compare time relating to the exact same WU. For instance, I'll let a WU crunch to 25%, then make a change and watch the time remaining go up/down. Typically, by 25% the time remaining is pretty stable and counts down 1 for 1. I focus more on the time remaing than on the actual time to complete a WU. I just don't want to get involved with backing up a WU to test later with different configurations.

I also compare by watching the PPD. It's a fairly good indicator except that it lags badly in terms of giving credit when a WU is complete. If I use PPD, I use the average PPD to see if it is increasing/decreasing over a period of days.

About 21 hours ago I tore apart my linux machine. I moved the GTX 275 to a new machine with less RAM but same CPU. This thread was originally created because I was getting just 13kPPD from the card with linux. Right now the machine has earned 44,947 points for the day and it has 3 hours to go! I have not yet started crunching for WCG on the machine yet. I'm going to give this a few days to see what kind of PPD I can get average before I add in WCG. The same model card on my HTPC machine is getting just 35kPPD average. If you want to watch my new windows machine go, you can find it here.

There was something very wrong with the linux machine based on the PPD. Here I am less than 1 day of crunching and my "average" as shown on gpugrid.net is already 1/3 of what the linux machine was after 50 days. It looks like I should be able to get some pie without even including my GTX 480 now. Woohoo! All I need to do is find some more cheap GTX 275+ cards to add to increase my crunching by even more. They'll be dedicated to crunching, so I don't really want to drop a load of dough on new cards. Maybe I can find some used cards for cheap in the forum somewhere.

[Kingcarcas]

I have been trying to find the sweet spot as well, ~2K F@H SMP/day if i don't run GPUGrid.

F@H is down to ~1K with GPUGrid, 10K/day GPUgrid, i lowered Boinc to 70% cpu and there seems to be no change, going to try 50%.

What i do know is i want more poweeerrr I'm just glad to be crunching, had to sell my 9800gtx+ that no longer got any WUs to buy a GT240 ($30 after MIR)

Win 7 x64, not going anywhere else. Been there done that.