Sam's long crunching journey accross the crunching valleys

[EtaCarinae]

Sam has entered the WCG top 10 members club. Congratulations from all the team.

He has now entered the Valley of Tears. Tears because you can crunch for a long time and nothing happens.
This is a long long period were there will be no more rank improvement. The next step or 9th position is to pass "neu-innova". By chance this cruncher stopped crunching altogether so he set up a fixed milestone. But the distance to the end of the valley of tears is 804 Million points away. This is about 75 days at full power or over two months if GPU WU last long enough, but there is a good probability that it will be so.

Then you enter the Valley of Fear. Fear because it is a valley that streches as you move as the exit is a fast moving target, "hillenbr". But nevertheless you see that the exit slowly becomes nearer. To get to the exit you need an additional 70 days again and only if GPU WUs last so long, and here it is really not a sure thing (that is the main fear). If this does not happen then you are stuck there whatever your efforts. Fast moving predators behind may then catch you.

But suppose you exit that valley and get at rank 8. You will then enter the Valley of Desperation. You will have no more fear from predators, they will be far behind.
Desperation because you will probably finish stuck in moving sands at the bottom of this valley.
The exit is at an additional 384 days or over a year of crunching full power. And at that time horizon your fuel (GPU WUs) will be exhausted.

But there is always hope. A new project with GPU capability may fall from the sky, maybe, who knows. You would then restart the engine at full power.

If this happens then you will have a very long assured path to the 4th position. After that you will enter the Valley of Eternity. The exit is a mirage in the very very far distance. As you move you have the feeling that the distance to the exit increases. Three gigantic statues of granit stand as guardians in front of that exit. Their mythical names are IBM, Sony, Marist. One of the guardians IBM is much taller and is known as the creator of the known crunching world.

If you are so brave as to increase much much more your speed, and be able to pass first the two stone guardians and finally topple the creator then you will enter the valley where nobody has been before. It is the mythical Valley of the Origins.

Good luck Sam may the force be with you.

[stoneageman]

It's the "Fast moving predators behind" that worries me
It's great to be representing the team in the top ten but of course that's not the whole picture as I'm way behind several other members regarding total run time. There's usz, XS_fallwind, trn-xs, RSmura, WindForce & of course ddtung who have racked up some amazing hours over the years.

[Rob_B]

Top 100 on run time is great no matter how you look at it. But in the end the returned results count. And you are in 10th place there too.

[[XC] serlv]

Love the story, Sam. I can relate to some of it. Somewhat.
Someone should have PMed me about GPU crunching. I would have jumped on this long before less than a week ago.

After I came down with the very thing I crunch against I never got back up to anything close to full crunching power. serious illness and escape from it has a way of pulling your attention elsewhere.

For several years I was #2 on this team, behind Victor. I went from crunching 25 single core rigs, to 18 or so dual cores (or DPs), to about 15 quads. Then I got ill and had no income for a while. I had to take a long ( actually a quick one... ) at the sanity of $700+ power bills, considering my situation. When I staarted to get healthier I started crunching a little more than the one rig i left on while laid out. I decided no more than 5 or 6 crunchers. But seriously, I never even got there regularly. At best it was like 3 or 4, usually two.

so I went from #2 on the team, and #16 globally to like maybe in the top 25 and #125 globally, but not in RAC or daily production. Still something to be proud of. I crunched my ass off, for many years.

What I want to know is WHY DIDN'T ONE OF YOU MOFOS pm me and tell me about GPU crunching. Damnit! sheesh.

Seriously though, it is good to see what these vid cards have done for the project and this team. Tis my bad, I haven't been around much for quite a while. but damn, I sure wish someone had let me know about WCG GPU work. Hello, Dave... I could have been running 5 or 6 rigs with good cards. Blows my mind what some people are putting out with them.

[Movieman]

Scott my friend: Because I knew you, knew your situation and didn't ever want you to feel any pressure at all to do something I wasn't sure you could afford.
I also wasn't too sure of your health. Was the cancer gone or were you just giving me the "stiff upper lip" like the Brits do. I just didn't know and my friend, I was so happy to hear from you that the surgery went well..That I can't put into words..
All I wanted to see from you is one machine, any machine, on the team and that you did..When it stopped was when you heard from me..

[[XC] serlv]

Dude! Just pretty much pulling your chain. Didn't mean to throw a guilt trip. There is none. As you said, my priorities were elsewhere. And so was my head. Looking at it now, it is easy enough to say I wish I did this or I wish I'd done that, but that is crazy. I wasn't even on the forums at all for the longest time.

I'm just happy to be here. That is the bottom line. The rest is all gravy. Many people don't come out of it so good. Looks like you're stuck with me!

[Movieman]

Scott, beleive me when I tell you that I'm just happy that your here too..Had days I wondered if we were going to lose you and that couldn't happen.
Your one of a kind my friend..
Now tonight I wish you were here..There's close to $12000.00 in hardware spread on my kirchen table,,

[[XC] serlv]

Wow. Gonna be some rig!
Hey did I tell ya I'm coming east in the last days of August? Unfortunately, I'm not going to make it to NH. Going to Philly ( did you know I was born there? } for a convention. I'm going with the missus ( oh wait, I'm not married ).

[EtaCarinae]

[XC] serlv, it is never too late. We do not know really when this GPU crunching at WCG will stop. We have to reach 100 million ppd as a team. Your additional cards are welcome.

[EtaCarinae]

Top 100 on run time is great no matter how you look at it. But in the end the returned results count. And you are in 10th place there too.

Allow me a large and long digression on this thread to comment the above post.

There is a direct correlation between results and points in a given project. If you crunch with a GPU the number of results increase because you process them much faster. You get also more points because again you have more processing power devoted to the taks. This is not an illusion, the GPU additional power is real processing power so it is correct and fair that compared to a CPU only crunching, you get more points and more results with CPU+GPU.

Across projects it is very different. If a project has WU's that take say 5 hours to process on a given machine, and another project on the identical machine has WUs that are processed in 1 hour, then both machines will get the same number of points but the number of results for the same amount will vary fivefold. This was true with HCC before there was GPU computing. The HCC WUs were already shorter than the ones of the other projects. The problem remains except if WCG decides to normalize the size of the WUs on a reference machine so that all projects on that reference machine have WUs that take the same processing time. Unless such a calibration is done the only real reference measurement between crunchers is the total number of points that reflect the real work that has been effectively devoted to the grid.

What about runtime. This is a philosophical question and will come back to this later.
Let's compare two machines with two different CPUs, all crunching the same project. The first machine has a very slow and inefficient CPU (say 1 core for sake of having it simple) and it runs 24/7, everyday. This will add 1 day runtime to the runtime score of this machine.
Now I come and put on the grid again another 1 core CPU, running 24/7 but extremely efficient and all in all 5 times faster than the previous one. Both runtime scores will be absolutely identical. But the faster CPU in the same timeframe will have produced 5 times more work, that is five times more results and points. Both crunchers have devoted the machine they have available fully but one has done five times more work than the other.
Is runtime a fairer approach. Not really in fact. First because a powerful cruncher will devote not one but many machines and with a maximum number of physical cores. So he will anyway get many runtime days per day. That is perfectly normal. An institutional cruncher, like a bank, a university, a large corporation etc... will devote thousands of low power cores. They will produce tremendous runtime scores (but not a volume of work in relation to the runtime score), and the individual cruncher stands no chance against them on this type of ranking.

Now comes the philosophical question. Which system is more fair. For me none in reality. The only absolutely fair system is no system at all. Total anonimity in terms of contribution. No rankings at all, no measurement at all, no scores, no badges, nothing. A cruncher is a cruncher point. Perfect equality. A cruncher is absolutely identical to another one, be it IBM or one contributing 1 WU per day.
This is a utopian answer. It may suit a few but the the human character being what it is, such a system will not work and will attract very very few people. To be attractive the evaluation system in place must satisfy the following aspects of the human character: ego, power, competition, reward etc. etc. Some of you will not like what I say but so it is and if you can satisfy at least partly those character elements then you may have some success.

Taking into account what I said before, what is then the best unit to measure, points, runtime, results.

1) The scientist. He wants the project done as quickly as possible and get the results as fast as he can use them. Projects cost, and the longer they take the more they cost. So for the Scientist points is the best ranking system to motivate powerful crunchers to put more and more power and to attract them by rewarding the crunching power. So it is points.

2) The powerful cruncher. Easy answer he wants points as the measurement system because it will favor him.

3) Institutional crunchers. They crunch for various reasons and for them runtime is the best unit, because it is the only element they can communicate in terms of advertising. Points mean nothing and results it is difficult to evaluate. If you advertise that you have given 1'000 years of computing power, this is easy to grasp because we mere mortals live in average between 70-80 years.

4) The less powerful cruncher (small CPU and no GPU). No system is completely fair for him unfortunately, but the fairest one is based on runtime ranking. The teams approach allows a low power cruncher to be in a team and through a collective effort improve the rankings.

5) WCG. They must address everybody and need averybody. This is why three rankings exist, runtime points and results. And then we have badges that are based on runtime and not points. The whole range of possibilities allows each cruncher to rank itself the way he prefers. Badges are very motivating for those who crunch with little power, because it is very challenging to get all of them. WCG is right to have that system because it is attractive to the large majority of the crunchers. WCG needs projects to be done in an acceptable timeframe, to satisfy scientist and to attract new projects. Powerful crunchers are also required and points is for them.

Results is nice to have but it is weakly correlated with the real amount of work that is done. The size of a WU is artificial and can be decided for each project. So the number of WUs crunched is not a reliable standard.

To conclude points or runtime are in the end the two best measurement systems.

[[XC] serlv]

Thanks for the encouragement. Been trying to decide if purchasing another card or two is worth it. I hear talk of 80 days or less in terms of GPU HCC, etc. I'm not sure if I'm even getting all that I can out of this one 7850, either yet. For instance, I seem to be getting less than 20, 000 points a day more than with just the CPU's alone. I thought it would be doing better than that. . Yeah 45K-55K is better than 30K, alone, but I'm not sure it is worth it in return for dollars spent. I think I expected bigger yields on this "test" installation. I may be doing something wrong and am definitely open to suggestion. I'd like to get that settled or optimized, first, before I further delve in. So I know what I'm dealing with and what I can expect. Then I can make a clearer purchasing decision.

[Snow Crash]

The "number of results" is extremely relevant and has a very important place in the world of crunching. The size of a WU and hence the number of results produced over a given period of time is much more reliable than PPD when comparing 2 machines. How quickly one can process similar WUs with varying application optimizations along with hardware / software configurations is very useful for figuring out "how much can I crunch?" and is a measure we here at XtremeSystems use continually. The "speed" at which we process WUs serves the altruistic goal of "get the results back to the scientists" and also is the best way to feed my inner point whore :-)

Another measure used by some other projects is to recognize individuals for exceptional results produced by WUs they processed ... for example finding a pulsar or the largest prime. I've also seen people recognized for processing the most energy efficient protein on a project within a given time frame. WHile WCG currently dos nothing of that nature, I thought it would be worth mentioning as part of a "motivational" discussion.

[Snow Crash]

Thanks for the encouragement. Been trying to decide if purchasing another card or two is worth it. I hear talk of 80 days or less in terms of GPU HCC, etc. I'm not sure if I'm even getting all that I can out of this one 7850, either yet. For instance, I seem to be getting less than 20, 000 points a day more than with just the CPU's alone. I thought it would be doing better than that. . Yeah 45K-55K is better than 30K, alone, but I'm not sure it is worth it in return for dollars spent. I think I expected bigger yields on this "test" installation. I may be doing something wrong and am definitely open to suggestion. I'd like to get that settled or optimized, first, before I further delve in. So I know what I'm dealing with and what I can expect. Then I can make a clearer purchasing decision.

Are you running multiple concurrent WUs on that 7850? If you have that working well, don't crunch too many CPU WUs as they will starve the GPU.

I crunched 16 at a time with an i7-920 + 7850 so I'm sure your DP rig can handle the CPU portions just fine. I have an XFX card that can go to 1325 if I pump up the volts to 1225.

If you post where you are at now for configuration I bet we can help you get better numbers (closer to 75K)

[Rob_B]

Allow me a large and long digression on this thread to comment the above post.

............

To conclude points or runtime are in the end the two best measurement systems.

I didn't mean to say one of the measurements is more important or better that the others. And I don't think you understood it like that.

Like you said, the difference in wu's between projects make the number of results returned not the best measurement to compare crunchers.
But it is nice to compare anyway.

The important thing is that the work gets done. And comparing your work with others made you work just that little harder. Like most of us I'm not here for the points or badges, but I do check the stats every day. It adds that little extra.

[RAMMIE]

Whatever the motivation that compels people to invest time and money into something that probably won't pay any dividends in their lifetime truly amazes me.

There is hope for us!

[[XC] serlv]

Are you running multiple concurrent WUs on that 7850? If you have that working well, don't crunch too many CPU WUs as they will starve the GPU.

I crunched 16 at a time with an i7-920 + 7850 so I'm sure your DP rig can handle the CPU portions just fine. I have an XFX card that can go to 1325 if I pump up the volts to 1225.

If you post where you are at now for configuration I bet we can help you get better numbers (closer to 75K)

Thanks for the offer, I appreciate it. If I can get this crunching where it "should be", then I'd feel better about springing for another card.
I'm using Old Chaps' app_config.xml. I'll post it below. It's basically 16 GPU units ( <gpu_usage>0.0625, <cpu_usage>0.25)

The processors are 3.16 GHz Harpertowns, not OC'ed.

The card is a Sapphire 7850, 2 gig OC Edition.
Not overclocked much, don't really know what to use. Am just using stock Catalyst Overdrive.
Should I be using other software?
In Catalyst's Overdrive the limit is 1050. I haven't been there yet. Was running at 1020 for a bit, but noticed the system was having a whole bunch of memory faulrs, Traced that to AVG, so I dumped that. Faults fell to nothing. Meanwhile I had turned it down to barely over stock. I'm probably ready to give it another shot.

I don't know much about overclocking a vid card, so I could definitely use some pointers.




File I'm using I got from OC's post.

<app_config>
<app>
<name>hcc1</name>
<max_concurrent>16</max_concurrent>
<gpu_versions>
<gpu_usage>0.0625</gpu_usage>
<cpu_usage>0.25</cpu_usage>
</gpu_versions>
</app>

<app>
<name>beta</name>
<max_concurrent>16</max_concurrent>
<gpu_versions>
<gpu_usage>0.0625</gpu_usage>
<cpu_usage>0.25</cpu_usage>
</gpu_versions>
</app>

</app_config>

[nanoprobe]

Now tonight I wish you were here..There's close to $12000.00 in hardware spread on my kirchen table,,

Wouldn't be spreading that news around. People know where you live.

[Movieman]

Wouldn't be spreading that news around. People know where you live.

People also know that I carry a gun and can hit what I aim at..

[OldChap]

Scott, a few observations for you:

Looking around at others' views on GPU overclock voltage I concluded that the maximum for our purposes was in the order of 1250mV provided that all elements of the card such as voltage regs. as well as the gpu itself stay at a reasonable temp. To monitor these I have used HWInfo and Gpu-z but others may suggest something better.

Sapphire Trixx looks real clunky but for me has given the control that others fail at in that it allows me to go higher in both frequency and voltage adjustment.

The pic below shows the slider that gives you access to voltage control



My personal view on getting these cards up to speed for crunching is to select the maximum voltage you are prepared to test at ...say 1220mV, and set that, then increase the core speed in steps of 25, letting it crunch for a couple of hours and observing if you are getting any invalids in your results. Increase the speed until you see invalids then drop back 25.

Crunching, my card will go a lot higher than say heaven benchmark so there seems no point in using anything other than crunching to test.

Now to the bit that may be contentious....

When running a 7950 in a q6600 rig I get around 90-100K ppd which in my view is about right for the difference in cpu speed (3.3 and 4.7) compared to a 3770K when I get ~130K. Running a 5870 in the same two rigs gave me the same results with both...25K ppd

This q6600 will happily run as many concurrent wu's as I want and I have tried 48 with no problem in fact running a large number, I am running 36 just now, seems to help with the offset staying relatively stable.

Saturating the cpu in this way does not seem to adversely effect runtime to any great degree. A freshly adjusted offset was giving me an average time for a wu at around 51 seconds with just 16 wu's. 24 hours later that was likely to be in the order of 53-54 seconds. moving to 36 wu's has added about a second at the start but no change after 24 hours.

Of course the catch with the information above is that it relates to me using a 7950 which is not the card you have. I hope though that it may open your eyes, and those of others here, to some possibilities.

Hope you can glean something useful from this.

[Snow Crash]

lol ... I type to slow ... nice post OC

I don't know much about overclocking a vid card, so I could definitely use some pointers.

Sapphire Trixx works pretty good for OC (and I have a Gigabyte card)
Once that is installed I got rid of the Catalyst stuff that was not the driver itself.
I'll tell you about the silly quirks first ... not big deal but confusing if you have not seen it before
1. There is a poorly shaded scrollbar on the right, use that to get to Volts
2. If you wait to long to "Apply" your settings, sometimes it only remembers the last specific setting and not the entire set so I always change voltage first
3. If any of the sliders don't go as far as you want them to try setting as high as you can and then click Apply, sometime the slider "grows" some more

I'm running my GPU at 1325, MEM at 1175, Board Power +20, mvolts 1225.

I always start with upping the volts and board power first.
Next I turn down the memory and up the GPU.

So now it comes to testing ... well, WCG HCC is the best test so just let that baby rip and make sure to check your results on their website to make sure you are not getting "Invalid" returns. When I test a new OC, I suspend all but the running WUs and then sort them by deadline which puts the re-runs at the top. I then unsuspend the re-runs which are usually already waiting for a return so you can see a "Valid" (or not) returns quite quickly. A small number of isn't going to tell you truly how stable you are but if you get any "Invalid" after say the first 32 or so you need to up the volts or lower the clock. If everything looks good, unsuspend the rest of the WUs and check in now and then at WCG to make sure it's still all good. Once I get 24 hours with no "Invalids" I consider that stable !

[OldChap]

I would be tempted to say "Great minds...." but it is encouraging to see that you use essentially the same process SC.

Incidentally, for those wanting to play around with the number of wu's running I am running Boinc 7.0.55 which lets you change on the fly by using the Advanced > Read Config File in Boincmanager. I see good things now when I do not try to match the runtime with the offset. Average runtime for a wu of say 50-55 secs and a separation of ~35 to 40 secs works well for me.

EDIT: Just to acknowledge the OP, I heard that SAM is dedicated and I can see it in the results but did you know he runs a q6600 on cpu only due to small power supply....all that crunching power and he STILL wants the last little bit available to him.

[nanoprobe]

People also know that I carry a gun and can hit what I aim at..

Duly noted.

And a big for SAM. Your dedication is much appreciated by all.

[[XC] serlv]

All this info sounds good. Will be trying it. Appreciate the time it took to share your experiences and learned wisdom.

I have a question for you all.

When my machine is in full crunching mode and I go to use it, say get on the web or whatever, oftentimes BOINC seems to have the complete attention of the system. It will be "in the grip", almost completely unresponsive to user input. I'll tap the Windows key and oftentimess, literally minutes later, I'll get a response. At that point I can snooze BOINC or open up a browser.

Does this happen with other people's rigs? When I used to run a lot of rigs I never noticed that almost complete unresponsiveness. BOINC would seem to gracefully allow the user to use the rig when he wanted, almost immediately.

[Movieman]

All this info sounds good. Will be trying it. Appreciate the time it took to share your experiences and learned wisdom.

I have a question for you all.

When my machine is in full crunching mode and I go to use it, say get on the web or whatever, oftentimes BOINC seems to have the complete attention of the system. It will be "in the grip", almost completely unresponsive to user input. I'll tap the Windows key and oftentimess, literally minutes later, I'll get a response. At that point I can snooze BOINC or open up a browser.

Does this happen with other people's rigs? When I used to run a lot of rigs I never noticed that almost complete unresponsiveness. BOINC would seem to gracefully allow the user to use the rig when he wanted, almost immediately.

I don't see this..On my dualie 2011 rig I have a 5780 card running 3GPU WU and 13 cpu wu with HT shut off.
I use it as my daily driver and I do see hesitation when playing a video but no other issues.
Now this machine is a lot gigher up( on cpu and memory but not on vid) than the Harpertown but still that machine should be fine.

[OldChap]

I wonder if this is gpu or cpu related???

I can't help feeling that you might have an older card sitting in a drawer somewhere...how about you add that and run your desktop from there..... just as a means to find the source of this issue which, incidentally, I have seen before but my memory is not filling in the blanks The above is what I think I did but.... you know how it goes.