Optimizing # ATI GPUs Crunching for Best Production Efficiency

[jeanguy2]

Hello fellows XS members,

I always wanted to confirm the best way to configure my rig to maximize ATI GPU production using the app_info file method. Today?s change from ATI GPU hcc 6.56 to ATI GPU hcc 7.05 gave me the opportunity and the catalyst required to run some testing. You will fine my rig?s details below in my signature. I am running BOINC v7.0.28 and using Bun-Bun?s revised ATI GPU only app_info file that is located in the first post of the app_info thread in this forum. All timings were obtained while running exclusively GPU threads. I used an average of their completion times in the BOINC Manager and crosschecked it with my stopwatch. Since I have a radeon 6990 I elected to run 2, 4, 6 and 8 ATI GPUs concurrently to see which set up would produced 24 completed ATI GPU hcc 7.05 WUs first.

# WUs | Run time | # Periods for 24 WUs | Production time | Time differential
2 -------------- 2:58 ----------------12 -------------------- 35:36 ------------- +13.36
4 -------------- 4:23 ----------------06 -------------------- 26:18 ------------- +04:18
6 -------------- 6:20 --------------- 04 -------------------- 25:20 ------------- +03:20
8 -------------- 7:20 --------------- 03 -------------------- 22:00 -------------- BEST

In summary, for my rig, it appears to be more efficient to run 8 ATI hcc 7.05 WUs concurrently (4 ATI WUs per GPU).

NB: Remember that the WU completion time in BOINC manager will only be accurate if you have selected 100% of CPU time in the preference options. Always use a stop watch to confirm.

FYI,
Jean-Guy

[0ne.shot]

Can you try running .1 GPUs and .8 nCPUs? I think this should give 10 CPU cores and .1 GPUs.

[jeanguy2]

Hello One.Shot,

The grand children have joined us for the weekend and I will also be involved with the local celebration of Rememberance Day.

I acknowledge your interesting above suggestion and I will run it ASAF. Likely Monday evening or Tuesday morning next week.

The 6990 has two GPUs on one board. A 0.25 setting gives me 4 x 0.25 GPUs per GPUs for a total of 8. This matches the 8 CPUs available on the 2600K with HT. To get a total of 10 GPUs I will have to input 0.20 GPU which should give 10 GPU total (5 per GPU). This would match the 0.8 nCPU setting you suggested (10 CPUs). Let me know if my maths are wrong.

I will hence edit the initial post and add the additional results with your suggested #s.

Have a great Rememberance Day weekend,
Jean-Guy

WE SHALL REMEMBER THEM....

[0ne.shot]

Ok. I couldn't wait and did the test with .8 Average nCPUs and .1 GPUs. This runs 10 HCC GPU WUs simultaneously.

----------------------------------------------------------------------

WUs with 10 CPUs and .1 GPUs

Total time for 20 WUs was 5632 seconds or 93min 51.6 sec

Total time for first 10 WUs was 2868 seconds or 47min 48 seconds

Average WU time in first set was 286.8 seconds or 4 min 46.8 seconds

Total time for second 10 WU set was 2764 seconds 40 min 4 seconds

Average WU time in second set was 276.4 seconds or 4 min 36.4 seconds

Overall average time per WU was 281.6 seconds or 4min 41.6 sec

-----------------------------------------------------------------------

WUs with 8 CPUs and .125 GPUs

Total time for 16 WUs was 3980 seconds or 66 min 20 sec

Total time for first 8 WUs was 1956 seconds or 32 min 36 seconds

Average WU time in first set was 195.6 seconds or 3 min 15.6 seconds

Total time for second 8 WU set was 2024 seconds 33 min 44 seconds

Average WU time in second set was 202.4 seconds or 3 min 22.4 seconds

Overall average time per WU was 248.75 seconds or 4min 8.75 sec

--------------------------------------------------------------------------------

Test system is a 2700k @ 4.7 GHz + 7950 @ 1025Mhz core + 1600MHz RAM

[bluestang]

If running more WUs than threads, make sure you try and offset the WUs. Even though they will eventually sync again, it will be more efficient.

[jeanguy2]

Bluestang,

Can you please elaborate on your settings?
How many ATI GPUs on your gig 7950?
How many CPU thread are you using?
What are your settings in app_info for that host? (GPU & nCPU)
How many returns does in generate daily?
What is your average completion time per WUs? How are you measuring the WU completion time?

Thanks,
Jean-Guy

[Johnmark]

Jeanguy2, have you tried slowing down the CPU while running 8 GPU W.U.'s, only? If the CPU is not running 100% wouldn't slowing it down reduce power consumption while maintaining the same overall output?

[haschioz]

no output will not be the same because the gpu sits idle as the cpu crunches, if however the wcg techs could assign cpu as needed and not fixed as it now (one gpu thread per cpu thread) we could quite a bit better performance. but i doubt they will do that as they got their handsful with the masses of wus already. but powerconsumption should go down a little. when you havent overvoltaged the chip its going to be negligable as the cpu draws more power but for a shorter amount of time (increase in power is linear to increase in clock speed-up)

[jeanguy2]

Jeanguy2, have you tried slowing down the CPU while running 8 GPU W.U.'s, only? If the CPU is not running 100% wouldn't slowing it down reduce power consumption while maintaining the same overall output?

Haschioz comments are spot on... I have actually done the opposite, I raised my CPU speed from 3.8 GHz to 4.5GHZ. This caused a modest wattage increase but yielded significant performance increase. With 8 WUs running on my 6990 timnig were/are:

CPU @ 3.8 GHz - GPU 880/1250 = 17:40 (CPU 5:30 & GPU 12:10)
CPU @ 4.5 GHz - GPU 950/1250 = 14:10 (CPU 4:15 & GPU 9:55)

I am letting the 4.5 GHz config run for a couple days to insure I do not get invalid returns. My next OC step wiil be CPU @ 4.8 GHz and 6990 @ 1000/1250.

The 6990 is not the best crunching card, 4 WUs per GPU appears to be my limit. If more than 4 WUs per card attempted, (error code 233) start showing up. Newer video cards 7xxx likely benefit even more from increasing the CPU and GPU overclocks.

[Johnmark]

Haschioz comments are spot on... I have actually done the opposite, I raised my CPU speed from 3.8 GHz to 4.5GHZ. This caused a modest wattage increase but yielded significant performance increase. With 8 WUs running on my 6990 timnig were/are:

CPU @ 3.8 GHz - GPU 880/1250 = 17:40 (CPU 5:30 & GPU 12:10)
CPU @ 4.5 GHz - GPU 950/1250 = 14:10 (CPU 4:15 & GPU 9:55)

I am letting the 4.5 GHz config run for a couple days to insure I do not get invalid returns. My next OC step wiil be CPU @ 4.8 GHz and 6990 @ 1000/1250.

The 6990 is not the best crunching card, 4 WUs per GPU appears to be my limit. If more than 4 WUs per card attempted, (error code 233) start showing up. Newer video cards 7xxx likely benefit even more from increasing the CPU and GPU overclocks.

haschioz made a good point on how the cpu is utilized, didn't realize that. Also for some reason I'm getting conflicting information on how much CPU time is being used. eg the desktop CPU usage meter is reading between 40-70%, task manager shows all 8 threads running at 100% and Asus's monitoring tool shows each thread using about 40% ?

Your running 8 wu's now and getting your wu's completed in 9:55 using a 6990! And I'm running around 9 min a wu running 8 wu at a time (GPU only) using a Asus 7950 running at 1000 mhz and the same cpu as you clocked at 4400mhz.
It will be nice to get a couple full day's of run time on this rig as the highest daily output according to WCG has only been 580k points. Thinking it should be closer to 750k

[OldChap]

If running more WUs than threads, make sure you try and offset the WUs. Even though they will eventually sync again, it will be more efficient.

Just an observation on this from me:

I have run 8, 10 and 16 on my 7950 during running before the pause and now running 14 on 7 threads .... It takes a while to set up but attempting to get MOST of the wu's starting around 40-50 seconds apart does show that times for MOST of the wu's are reduced .... there may be up to 1 minute difference between each.

Perhaps more to the point, I check up on that rig each day and see that MOST of the wu's remain separated in time day by day. They do not all revert to starting together as happens when trying to run >1 on each 5870

[jeanguy2]

JohnMark,

You mis-interpreted my numbers:

CPU @ 3.8 GHz - GPU 880/1250 = 17:40 (CPU 5:30 & GPU 12:10)
CPU @ 4.5 GHz - GPU 950/1250 = 14:10 (CPU 4:15 & GPU 9:55)

The numbers in red are the total average time needed to complete one WU (running 1CPU + 0.xx ATI GPU") when running 8 concurrent WUs on my 6990. So with a CPU @ 4.5GHZ - GPU 950-1250 the total completion running time for one GPU thread is (4.15 CPU + 9:55 GPU) = 14:10

Your single engine solo jetset 7950 is flying circles around my 6990 twin engine fabric biplane. If your total average production is inded 9:00 ( ATI GPU + 0.125 CPU) minutes then you best the 6990 by 5:10 (14:10 - 9)

I sure hope Saint Nicks bring me one of them 7970...

[jeanguy2]

OC,

I am having problem (nothing new there...) visualizing this statement:

I have run 8, 10 and 16 on my 7950 during running before the pause and now running 14 on 7 threads .... It takes a while to set up but attempting to get MOST of the wu's starting around 40-50 seconds apart does show that times for MOST of the wu's are reduced .... there may be up to 1 minute difference between each.

Can you express it differently and specifically denote the # of CPU and/or (GPU + CPU) threads you uses by posting either your <avg ncpus> and <count> settings and or the the actual data in the BOINC manager task page [ie: "running 1CPU + 0.xx ATI GPU"). The number "7" throws me for a loop..

Thanks,
Jean-Guy

[OldChap]

If you go back to the calculation concept I added to the app_info thread:

<avg_ncpus>0.8</avg_ncpus> <this number here = 1/number of threads you wish to run multiplied by the number of cores available so for 8 cores and 10 threads this would be 1/10*8=0.8
<max_ncpus>1.0</max_ncpus>
<coproc>
<type>ATI</type>
<count>.1</count> <assuming one card this number is 1/number of threads you wish to run so for 10=0.1

Now let us adjust this as follows:

number of threads you wish to run (14) so 1/14 multiplied by the number of cores you wish to run (7) we get 0.5

so:

<avg_ncpus>0.5</avg_ncpus>
<max_ncpus>1.0</max_ncpus>
<coproc>
<type>ATI</type>
<count>.0714</count> Nothing changed here as I want to run 14 instances on this

This should make it clear that at 0.5 cpu per GPU instance this is running 7 cpu threads leaving one clear to do purely cpu work


Does this help?

[jeanguy2]

OC, thanks for info can you check data provided,

You State:

Now let us adjust this as follows:

number of threads you wish to run (14) so 1/14 multiplied by the number of cores you wish to run (7) we get 0.5

As per your own formula, ncpu equals

<avg_ncpus>0.8</avg_ncpus> <this number here = 1/number of threads you wish to run multiplied by the number of cores available so for 8 cores and 10 threads this would be 1/10*8=0.8

The last portion of the equation is "number of cores available" not the number of core you want to use as stated in your first statement. So an ncpu of 0.5 would yield 16 threads ( 1/16*8=0.5) not 14 threads (1/14*8=0.571).

Therefore after setting:

<avg_ncpus>0.5</avg_ncpus>
<max_ncpus>1.0</max_ncpus>
<coproc>
<type>ATI</type>
<count>.0714</count>

You should see in the BOINC manager "tasks" page a total of 16 threads active/running. 2 of them CPU threads and another 14 (CPU + 0.0714 ATI GPU) threads

FYC,
Jean-Guy

[Johnmark]

Are the settings you just posted efficient ?

[Bun-Bun]

no output will not be the same because the gpu sits idle as the cpu crunches, if however the wcg techs could assign cpu as needed and not fixed as it now (one gpu thread per cpu thread) we could quite a bit better performance. but i doubt they will do that as they got their handsful with the masses of wus already. but powerconsumption should go down a little. when you havent overvoltaged the chip its going to be negligable as the cpu draws more power but for a shorter amount of time (increase in power is linear to increase in clock speed-up)

Hopefully for the next project that goes GPU they will optimize things a little better.

[OldChap]

OC, thanks for info can you check data provided,

You State:



As per your own formula, ncpu equals The last portion of the equation is "number of cores available" not the number of core you want to use as stated in your first statement. So an ncpu of 0.5 would yield 16 threads ( 1/16*8=0.5) not 14 threads (1/14*8=0.571).

Therefore after setting:

<avg_ncpus>0.5</avg_ncpus>
<max_ncpus>1.0</max_ncpus>
<coproc>
<type>ATI</type>
<count>.0714</count>

You should see in the BOINC manager "tasks" page a total of 16 threads active/running. 2 of them CPU threads and another 14 (CPU + 0.0714 ATI GPU) threads

FYC,
Jean-Guy

I will try to clarify:

The first is number of cores available but in all cases it is the number of cores you make available You are setting these numbers.

Secondly remember here we are setting up GPU only so that portion of the app_info shown above is setting parameters for GPU

Looking at the task manager page one finds 14 instances of GPU wu's in progress and a single CPU wu too for a total of 15.

each of the GPU wu's is showing as 0.5 CPU + 0.0714 ati

The CPU portion of the app_info has not been altered and therefore runs 1 wu per core.

Johnmark: I have not looked at how efficient this runs. The idea for now has been to clear the backlog of cpu work.

I shall look at results in more detail later tonight.

[jeanguy2]

OC,

Thank very much for the clarification. Once I get an HD 7xxx video card, I will set the above nCPU and counts parameter(s) to try them.

[OldChap]

Before the pause when I ran 16 instances on 8 cores I downloaded results pages (10 of them) to my spreadsheet

The numbers crunched out to an average of a wu every 41.7 seconds.

Today I picked another 10 pages of running 14 instances on 7 threads and the numbers are as one would expect almost the same at 41.4 seconds average.

I will try some variations over the next few days. Due to the way that WCG records completion times in hours rather than minutes these results are less than completely accurate but it should allow me to see a trend

[haschioz]

btw: how do you dl stats from your page? need to do that, i think my 7950 is faster with 8 wu and consumes less electric.

[OldChap]

Sorry, figure of speech.

Copy and pasted 10 pages into spreadsheet

Use of formulae like =1*LEFT(G2,4) and =1*RIGHT(G2,4) where G2 is in the claimed/awarded points column will strip the numbers out and make them useful

[Johnmark]

OK so I've got the 7950 and 7970 working pretty good(2600k CPU's @4800Mhz), THANKS to the TEAM.
Don't have any good daily production data as I've been playing around with those two machines a lot.

However as others with high end cards have no doubt found out, our CPU's are the limiting our overall production and I can't seem to find a way around it.

The problem is the amount of CPU time/resources a wu needs when it goes to finish up (not sure what it's doing during this period). The units complete so fast (1:42-3min) they bunch up at 49.707% complete and 99.707% causing CPU load to go to 100% and thus slowing down the whole process. Currently running 1o wu's, and it can take over 1 min to go from 49.707 to 50%.
I've tried offsetting the wu times by suspending a given unit at x% and letting a new one start, but I'm finding with the 7970 running at 1200 Mhz this just is not possible, they still all bunch up.

Anyone with a 7970 Beast been able to solve this ? What are the rest of you with high end ATI's running that may help to mitigate? Is there a sweet spot for the # wu's running ?

Inquiring minds need to know

[kjeldoran]

OK so I've got the 7950 and 7970 working pretty good(2600k CPU's @4800Mhz), THANKS to the TEAM.
Don't have any good daily production data as I've been playing around with those two machines a lot.

However as others with high end cards have no doubt found out, our CPU's are the limiting our overall production and I can't seem to find a way around it.

The problem is the amount of CPU time/resources a wu needs when it goes to finish up (not sure what it's doing during this period). The units complete so fast (1:42-3min) they bunch up at 49.707% complete and 99.707% causing CPU load to go to 100% and thus slowing down the whole process. Currently running 1o wu's, and it can take over 1 min to go from 49.707 to 50%.
I've tried offsetting the wu times by suspending a given unit at x% and letting a new one start, but I'm finding with the 7970 running at 1200 Mhz this just is not possible, they still all bunch up.

Anyone with a 7970 Beast been able to solve this ? What are the rest of you with high end ATI's running that may help to mitigate? Is there a sweet spot for the # wu's running ?

Inquiring minds need to know

With these new long WUs, I haven't found a sweet spot for offsetting them as that middle portion messes it up. To be honest, it will probably require higher clocked CPUs or a new CPU with better performance. Maybe haswell will give is something?

[OldChap]

I have been running 16 on 1 card and 1 3770K at 4.8 and they seem to Mostly stay spread out BUT I have also, it seems, been getting a few invalids so I will start again at 10 and increment daily in an attempt to get to the bottom of the problem