Build Log: 100,000 PPD by 1/23/2011

[trn]

Hi my name is TRN and i'm addicted to crunching. **HI TRN**

Anyways... I set a goal for myself, to average 100,000 PPD boinc by 1/23/2011. Why 1/23/2011? because I first joined WCG on 1/23/2010 and that sounds like a nice 1 year anniversary gift to myself

This thread will be a build log of my current project to build my computers into a proper mass-water-cooled farm and also upgrade my current collection with some new toys and a continuous build log to hit my goal. Right now I currently have machines scattered about randomly. Off the bat I'd like to thank the team; without the XS WCG team I doubt I would ever be this motivated to work on this project. I am inspired what many other XS users have done before me. I'm impressed with mass water cooling projects by; Fallwind, 123bob, and naja. And thanks to Jaco and MM for answering some questions i've had so far.

*Edit* This thread has been linked on www.poppageek.com

Current computers not going in the farm:
At my office (not in my farm) dual E5520 and 930 OC'd to 3.3GHz.
Main Rig at home: (won't go in the farm) 980x

Current rigs for da farm:
1: 980x B1
2: 920 D0
1: w3520 D0
7: L5640 A0

Some new arrivials, 7 L5640's posing with my second 980x. Mobo's will be Gigabyte x58a-UD3R's, except for one 920 which is in a cheapo asrock x58.


I bought two Car Rads; somehow I imagined these things being smaller even the 'smaller' one absolutely dwarfs my mountain mods extended ascension! I put a 120mm gentle typhoon on the bottom rad for a size reference. The fan on the top rad is a 14" car rad fan off ebay; I have 2; a pair fits perfectly on either rad. The fans are rather cheap, unbalanced and noise, but have good enough air output and the price was right. I tested them out today and they don't undervolt well, they really only run at 12v and the blades are unbalanced. The top Rad is from a v8 Chevy lumina or somthing like that, and the bottom one is from a 1990 Blazer, both are 34" in total width! I spent the afternoon flushing out the two with a garden hose, the bottom one which is quite beat up and dirty could still use some more flushing; I even pulled a rock about 1cm in diameter out of the plenium I think someone dug this one up in their backyard and put it on ebay.


I think I have settled on a spot to build the farm; in my garage with my rads exhausting out a window. I live in South Florida so the less heat I dump into my house then the less I have to negate that with air conditioning. I've been running a 920 and a Q6600 in my garage for maybe 6 months or so and even with FL heat and humidity they have held up, no signs of rust or excessive wear.

Todo list:

- See what these L5640 A0's can do... I got one on the test bench right now learning how to OC these. I have't messed with them too much but so far it seems the max multiplier on the A0's is 18x instead of 19x on B0/B1's. I'm trying to underclock the ram as much as possible; right now after 2 trys it seems that i'm stable at 200 x 18 but it took a QPI of 1.3v bios (with vdroop so I hope the real QPI is lower.) I OC with an EVGA board on my main rig so I'm not so familar with the Gigabyte bios's (*hint, any tips appreciated.) I also don't want to pass 1.3v QPI and burn out the A0's. at 1.28v QPI I was getting 124 code (not enough VTT) BSOD's.

- I think my biggest challenge will be supplying enough power to the farm, Cooling... no problem. I also just got a Kill-a-Watt; my best 45nm clocker is using 250W headless at 4.0GHz My worst clocker was pulling 280w. I'm sure the l5640's and 980x will do much better. So far with a basic GPU my L5640 on the testbed is pulling 205w @ 3.6GHz, not great but not bad (its an A0, I think they use more power than B0/B1's.) So with what I have so far thats about ~2,400W + some extra for pumps and fans. And i'm going to need more room to grow to meet my goal The breaker box is on the opposite side of my garage, i'm thinking about getting an electrician to see if they could install a 240v (or two) because a single 120v plug aint going get the job done (anyone that runs their rigs on 240v and knows anything about electrical work got any handy tips?)

- I need to find some metal shelfing that I can use to build my farm on.

- I plan to build some sort of over the window shroud thing out of wood that can keep the radiators away from the window and direct the air out of the window. (I think i'll use both radiators, they are roughly the same size as the window opening.)

- I've already ran a network cable through my attic to the garage so i'm set there.

- Seal up all the unused holes in the radiators and flush the dirty one out even more.

- Buy alot of WC gear; like I said I like Fallwind's WC setup, he has two pipes on each side of the rads, he uses one side and has pump intakes on it, the pumps cross a few WB's then dump into the other pipe which is the intake. Simple genius solution that doesn't involve complicated splitting or even a reservoir, just many standard WC pumps running in parallel to create the loop.

- I'm also going to try installing Linux on future crunchers that I'm adding to the farm. I've used *nix years ago so i'm one small step ahead of a newb. I'm going to be installing Debian 5 because it has a netinstall which means I can install only the bare minimum of what is needed and no extra bloat. I was convinced to try Linux for crunchers after reading Jcool's linux vs win for HFCC thread. It seems that (for all projects) linux does 10-15% more work at 5-10% less credit; screw the credit id rather do the extra work

Ok... so as if it didn't seem like I had enough work to do on this project already; typing it all out made it seem like much more Back to the test bench I go to work on these l5640 OC's !

[[XC] mysticmerlin]

I'm Subscribed!! Go get em

[pwnajer]

Might look at plastic shelving. Back in the day I would run caseless and was able to lay the boards on the shelves without fear of shorting.4 teirs with 4 boards per teir. And it is cheaper/liter than metal.

[lkiller123]

One year and you are making such a great devotion!

Here is for the next DDTUNG!

[Sparky]

Too bad you weren't closer I'd help you with the electrical side of things and save you some money there.

Sounds like quite a venture!

[shoota]

I can't wait to see this come to fruition. Are you close to Tampa by chance? I wanna come see it lol!

[Bobsama]

Subscribed!!

Anyways, I'd like to see what you do with this. What are you using for CPU blocks?

[Plan.B]

Holy shiznit! Keep those pics coming. Xtreme WCG! I love it!

[trn]

Might look at plastic shelving.

Plastic is my last resort, Last plastic shelving unit I bought at Home Depot was a real POS. I have a q6600 sitting on it no problem, but the 920 I keep on it I keep elevated; I had the back plate melt a static bag it was sitting on before. I think these 1366 backplates can get quite hot.

here is for the next DDTUNG!

No way... I've seen pics of DDTung's setup. Only way I will be able to even match a fraction of what he did is due to tech progression. In another 8 years the Intel Core I22 256 core cpu's will do better than 100,000 PPD with only 10W

I can't wait to see this come to fruition. Are you close to Tampa by chance? I wanna come see it lol!

Palm Beach County

What are you using for CPU blocks?

For the L5640's I plan to use Enzotech Saphire's: cheap, low restriction, and I won't need top of the line blocks to cool these at 3.7 - 3.8GHz

Holy shiznit! Keep those pics coming. Xtreme WCG! I love it!

Will do:
So far this is what I can get from the L5640's: QPI is at 1.3v with load line level 1 (so hopefully the acctual uncore voltage is less.) @1.3QPI Bclk 200 or 205 or 210 all ran equally as stable; I didn't see the need to push it past 3.8GHz.


My "bench" setup, looks suspiciouly like a dinning room table



And the Money shot! Headless Watts should be ~185 - 190w without the 'gpu' and delta jet turbine.

[Movieman]

This kind of devotion demands some help.
I just grabbed 3x1 gig of PC3-12800 GSKill to have as spares.
It's your's if you want it.
Just PM me your shipping info.

[Jaco]

I would lower the QPI multi 1 step .
3780Mhz QPI link is a bit over the top (maybe allows you to lower QPI/VTT)
What's your ram and uncore speed?

[fallwind]

Looks like the farm is shaping up nicely! Here's a tip on cleaning the rads: save yourself alot of time and frustration and take them to a local rad shop and have them cleaned in their acid tank. It'll come out 10x better than you can ever do at home, especially the insides. Used rads are always caked full of scale, rust and dirt. Then just follow up with a water flush at home.

I got lucky and the stock fans on my car rad undervolt very nicely. I have them running on the 3.3v rail on the PSU no problem and they only draw 20w total. 12v will give great cooling but the fans will suck up alot of power and the noise will be unbearable. You really don't need a ton of airflow so maybe consider something like the 200mm Antec Big Boy fans. They're quiet and have three speeds. I'm using 3 of them on the rad cooling my personal rig.

[D_A]

Would it work if you mount them horizontally and run passive? Just a thought.

[STEvil]

possibly would work passive with both of them. Even one is enough for all those rigs.

[Sam_oslo]

This looks like a really good and big project. Those car-radiators are really bit tho, literally.

That L5640 is doing really good in 1.2v low-volt action. The chip seams to have more OC-headroom, but the BCLK-wall on MB seams to be limiting it.

Since you have 7 of them, it could be a good idea to see how they can do on a MB with higher BCLK-wall. BCLK240+ would make these chips rock @4.5GHz+, and it would be interesting to see if some MBs can do it.

[ChaosAD]

Nice big project you have there. Massive crunching power But i think that you are going to get me into trouble, im already considering on building 2 new rigs just for folding

[jcool]

Wow, you sure ain't kidding around mate. Very impressive chip lineup you got there
When I read the title I thought you meant 100k WCG

Now, I've had around 8 L5640's here, and I should have kept them all to myself
Awesome chips for crunching. Best Performance per Watt ratio, aside from running stock X5650/60/70's on a Tyan or SM DP board maybe.

Mine were all B0, so I don't really know how the A0 L5640's will fare (I thought I had all the A0 Westmere-EPs out there, and I didn't see one of those... guess I didn't have them all heh), but I'm positive you'll get to 3,6Ghz (200x18) on all of them. Beyond that point it's down to luck and tuning skill/patience, but I found that running between 3,6 and 3,8Ghz is the sweet spot for power/performance on my L5640 anyway.

If you need any input or help on OCing those chips, watercooling, etc. just shoot me a PM, I'll be more than willing to help in any way I can

[Zloyd]

Simply WOW !! Cant wait to see it finished

[Kryja]

You're going to make a lot of us green with envy with this

Best of luck mate, I hope you achieve your target in time

[Tom128]

This thread is going to cause a substantial increase in PPD for all of XS once we see this finished and can say "oh man he just did that, what's the cost of another 6-core compared to THIS guy?".

[slaveondope]

This thread is going to cause a substantial increase in PPD for all of XS once we see this finished and can say "oh man he just did that, what's the cost of another 6-core compared to THIS guy?".

BTW Im in the market for a used 6 core any one go one lying around

Seriously

[Tom128]

This kind of devotion demands some help.
I just grabbed 3x1 gig of PC3-12800 GSKill to have as spares.
It's your's if you want it.
Just PM me your shipping info.

This is a good point actually. I have at least one and I want to say two old (and loud) 40Gig IDE HD's you are welcome to have. If you just need a drive for an OS and BOINC they would get the job done. I also have two old laptop hard drives I believe, not positive if I still have any adapters to let them run on standard IDE/SATA ports though.

That's all I think I have that could help with your builds (Mostly AMD or OLD Intel CPU's and DDR2 - not helpful), but if you would like a few HD's I would be glad to help, just PM me an address

[Rise]

Dak better watch out! You can't say he didn't try to warn you!

GL with the build trn, looks like a lot of fun! 1 question: are you married? my wife would cut my nuts off if I tried something like that...

[trn]

Thanks to everyone for the overwhelming support I did expect some interest with my thread but not this much

Most of you are right, my car radiators are HUUUGE! and total overkill. I think i'm going to try and run them both and go with overkill though. I plan on setting up my farm in my unclimate controlled garage in South Florida so I need to be able to deal with high ambient temperatures year round. My goal is 100,000 PPD so thats going to take alot of fire power; even more than I am building now so I need room for future growth in cooling. For 100,000 PPD with only hexacores (for example) that would take around 3,100W - 3,200W in ambient temperatures that can exceed 33c! I think id rather build the WC system once and build extra cooling overhead to meet my goal.

This kind of devotion demands some help.
I just grabbed 3x1 gig of PC3-12800 GSKill to have as spares.
It's your's if you want it.

Thanks for the help! much appreciated

I would lower the QPI multi 1 step .
3780Mhz QPI link is a bit over the top (maybe allows you to lower QPI/VTT)
What's your ram and uncore speed?

Sound advise, after running overnight I went back into bios and checked my settings, my QPI was set at 1.34V! higher than I wanted! I backed off the bclk to 205 which let me drop my QPI to 1.32V, with vdroop i'm hoping the accrual uncore voltage is under 1.3v. The UD3R seems have a bit of vdroop, i'm running at level 1 line calibration (some vdroop.)

Settings are:
Vcore 1.21875v (load voltages are usaly 1.2v but fluctuate down to 1.184v)
QPI 1.32v
uncore multiplier 12x with 6x memory multiplier
Vdimm 1.3v with timings relaxed to 10,10,10,27 or something similar to that

Looks like the farm is shaping up nicely! Here's a tip on cleaning the rads: save yourself alot of time and frustration and take them to a local rad shop...

Good tips on the Rads and Fans. The bigger of the two rads is filthy and beat up and the smaller one has some grease and goo baked on the outside. The fans I got off ebay are cheapo's that a million different sellers are listing (heres one for example http://cgi.ebay.com/ebaymotors/NEW-1...#ht_3646wt_951 I think I bought a pair for ~$50 shipped.)

And once again thanks for posting pictures and a build log of your own Mass-WC'd farm; If I never saw your build log and your amazing collection of hexacores I would have never thought this was possible to do. This is why i'm posting my own build log also; I was inspired by others and shown how its done by the pro's, maybe someone can learn from my build log.

That L5640 is doing really good in 1.2v low-volt action. The chip seams to have more OC-headroom, but the BCLK-wall on MB seams to be limiting it.

Since you have 7 of them, it could be a good idea to see how they can do on a MB with higher BCLK-wall. BCLK240+ would make these chips rock @4.5GHz+, and it would be interesting to see if some MBs can do it.

I'm afraid trying to run these A0's at that sort of Bclk would be hexa-suicide. A B0/B1 L5640 has a turbo multiplier of 19x (vs my A0's 18) and B0/B1's are suppose to be less fragile. To even hit Bclk 210 stable it takes 1.34v QPI, and Bclk 200 needs 1.3v. Bclk of 240+ would take suicidal VTT

If you need any input or help on OCing those chips, watercooling, etc. just shoot me a PM, I'll be more than willing to help in any way I can

Thanks!

This thread is going to cause a substantial increase in PPD for all of XS once we see this finished and can say "oh man he just did that, what's the cost of another 6-core compared to THIS guy?".

Good point; I was so impressed with these L5640's I just ordered another 3. while 7 is a lucky number 10 has a much more even sound to it!

This is a good point actually. I have at least one and I want to say two old (and loud) 40Gig IDE HD's you are welcome to have.

Wow, thanks! I do need some HD's and a 40gig Ide drive is just fine for Boinc!


I'm not sure if these settings are 24/7/365 Boinc Stable yet, but I moved rig #1 off the test bench and set it crunching headless, changed the delta out for a S-Flex.


And here is the headless wattage, a nice even 175w

[slaveondope]

So you could use IDE HD's? Got 2 30gb WD's (loud as well but run) if you need them pm me maybe even another IDE somewhere too.

Free of course