Build Log: 100,000 PPD by 1/23/2011

[trn]

You can still use su if you want to. The Ubuntu forums gods will try to rain thunder, torment and bannage on me for mentioning this (blow me, Ubuntu mods), as it's not "the Ubuntu way", but you can type "sudo su" and you're there. No need for root password.
You also don't HAVE to install everything from the repos, either. Compiling things yourself works just fine and you can either do it "old skool" and leave the fresh toys in whatever directory you like or compile into deb packages and use the package manager.

I was typing that paragraph about ubuntu just thinking "D_A is going to have something to say about this" The truth that I realized is that since these are just crunching only machines... none of my gripes about Ubuntu matters; as long as its stable and works then that serves my purpose. I also need something that is easy to admin for many machines and ubuntu server fits that criteria. If I get a Linux box up for an everyday web browser, a project to mess around with, desktop, non gaming machine (which is somewhere on my to-do list, near the bottom ) then I will use a different distro; something that suites my taste better.

Also while Linux has improved immensely since the stone ages when I last used *nix some i'm not a big fan of the fragmentation of some of the distro's like Ubuntu were your suppose to do things "the Ubuntu way" use the gnome ubuntu-desktop enviroment, use sudo, or whatever else. I'm more of the old school configurable linux fan; I was liking Arch and I used Gentoo when it was new and liked the build from source approach and configurability.

Oh, and the rest of my project is in limbo for awhile:
- Linux issues solved.
- WC loop is working just fine with only one sub-loop and no fans.
- I ordered fans (6) from newark when they only had 5 in stock so I may be waiting a while. The newark page said they would ship direct from the mfg or distributor or something. No word via email about an order processing or shipping info or whatnot. Their site quotes a depressing 10-15 business days for shipping (really? a web business can't ship a product in less time than that these days???)
- Waiting on my 'Taco' 1/35hp pump from grainger; then I have to tear down and rebuild the loop at least part way. I plan to install the pump in between the two rads were I currently have a 1.5" ID hose bridge because I have space there and it should be easy (famous last words.) I'm not expecting fast shipping from grainger, I would consider myself lucky to get the pump this week.
- I have 9 out of my 14 hexacore's up and running; 1 isn't OC'd and another only has a mild OC. Running that setup for this week without messing with them constantly ought to give me an idea of what to expect later. I'm not running a very point friendly WCG mix of projects right now though. Linux boxes are on CEP2 WU's, windows boxes are finishing up some C4CW WU's before going back to 'exclusive' DDT2 (which really means all projects because there are so few DDT2 WU's these days.) Looks like alot of HPF2 WU's in the que which suck balls point wise if they don't throw a computation error.

[D_A]

...
Also while Linux has improved immensely since the stone ages when I last used *nix some i'm not a big fan of the fragmentation of some of the distro's like Ubuntu were your suppose to do things "the Ubuntu way" use the gnome ubuntu-desktop enviroment, use sudo, or whatever else. I'm more of the old school configurable linux fan; I was liking Arch and I used Gentoo when it was new and liked the build from source approach and configurability.

...

I'll comment on this particular paragraph because I think you have a good point. The "Ubuntu way" is a crock. I'm a big fan of being able to do what I want to my system when and how I want to. If the "Ubuntu Way (TM)" doesn't let me, then I find a way to do it MY way.
While you CAN choose a variety of desktops instead of Gnome, it's the default. There are other install CDs/DVDs available with KDE, XFCE etc on them, but you have to hunt for them or mess around installing things yourself and switch, which is a right pain IMHO.
As for sudo, well there are some good reasons for it to exist ... but I have to admit to using su when I'm compiling kernels coz it lets me do stuff that sudo doesn't, at least not easily.

[Sparky]

trn is XP Pro a viable OS for this kind of thing? Not sure what XP's limitations are for what you're doing.

If it would work OK, PM me.

[trn]

I installed 1 XP x64 cruncher yesterday, and I plan on installing one more. Just those two so they can crunch WCG and GPUGrid at the same time. I'm happy to keep a mix of OS's for now. Some windows computers, Win7 and XP x64 and also some Linux boxes. I figure a mix is good; I can always crunch projects that are only available for Linux or maybe something is only available or for some odd reason actually runs better on windows. Of the rest of my rigs I plan to install one more XP x64 rig and then everything else from now on Linux. And maybe later if XP x64 is working good i'll convert a Win7 or two more over to XP x64 so I can always have some good host available to Crunch both WCG and GPUGrid on.

D_A, have you ever messed around with Ganglia? It seems a bit excessive for a WCG farm but might be a cool way performance monitor many linux machines. http://ganglia.sourceforge.net/

[D_A]

D_A, have you ever messed around with Ganglia? It seems a bit excessive for a WCG farm but might be a cool way performance monitor many linux machines. http://ganglia.sourceforge.net/

Hey! You leave my ganglia out of ... oh you mean the software package. Nope, never played with it.

I wanted to build a cluster for ages, still do, but I've put that on hold for the time being. This sounds like something that would be handy for that.

[trn]

I was thinking it might just be a handy monitoring tool for a grid. Looks like it takes system data from the individual nodes and uploads them to a single computer and creates graphs and whatnot of usage statics. Kinda pointless for WCG since load should always be 100% cpu and ram and disk usage aren't very important or changing.

[onex]

trn, have you thought about separating each intake to a different setup?
using the same pipe over few processors won't cool down the second and the last one the same, imagine the temps going 40's at the first then outgoing at 60 to the second and 80 to the third,
why not using a 1 to 3 junction pipe running 3 separate hoses 1 for each setup, that would supply cooler water to each processor and the outtake won't add the temperatures up which should allow the radiator fans to cool it much better.

you'll need an intake pipe with an exit to each MB and an outtake pipe running from the first processor block, taking the load from the second and the third in serial.

so you have the same temperature fluid running through them all.

[trn]

trn, have you thought about separating each intake to a different setup?
using the same pipe over few processors won't cool down the second and the last one the same, imagine the temps going 40's at the first then outgoing at 60 to the second and 80 to the third,
why not using a 1 to 3 junction pipe running 3 separate hoses 1 for each setup, that would supply cooler water to each processor and the outtake won't add the temperatures up which should allow the radiator fans to cool it much better.

you'll need an intake pipe with an exit to each MB and an outtake pipe running from the first processor block, taking the load from the second and the third in serial.

so you have the same temperature fluid running through them all.

Onex, your right the first cpu will warm the water a little and the 2nd and 3rd processors won't get quite the best cooling but with my mix of CPU's and clocks I've planned for that. Most loops will be:

CPU1: 4.2GHz 980x or 970. At only 4.2GHz 980's are 970's take very little to keep cool
CPU2 and 3: L5640 3.7GHz @ 1.2vcore which remain ridiculously cool under water.

So with this CPU mix and 32nm's which arn't being pushed that hard they will remain quite cool. Right now on my test loop CPU1 is a 970 @ 4.2GHz without the Vcore tuned in and its running at 61-63c on the hottest core with no fans on the rads, just running passively and dumping heat into my garage. So far so good.

[onex]

so are you running a triple loop as the earlier page picture shows or only a single setup? and if so, how hot is you'r 2nd and 3rd CPU at that loop or rather what are you expecting?

[fallwind]

Running 2-4 CPU blocks in series is not a problem. My farm has 3 blocks on each "sub-loop" and the second and third CPU's don't run appreciably hotter than the first. You have to remember that the water is moving through each block so fast that it doesn't have time to increase in temp very much at all, maybe 1-2C. Having 1 or 2 massive radiators helps here as well, it assures that the water coming out of the pumps is at ambient temperature. So CPU block #1 gets ambient temp water, #2 gets ambient + 2C, #3 gets ambient +4C(rough guesses) which is more than sufficient for the kind of clocks we run for crunching.

[onex]

You have to remember that the water is moving through each block so fast

yes, you are right, that is most correct,
water going up to 1200Lp/h or 20L a minute per pump on a 1/2" or 3/4" pipes shouldn't have much time to gather heat at such small (CPU block) area,
i just filled up a 1 Litter bottle to put in the fridge, at 10 seconds a litter or 6L per minute that was quite a nice flow!

[trn]

I'm glad Fallwind answered that because really i'm just copying what I've seen him do with his farm. And with these 32nm Hexacores they run so cool that the maximum best water cooling isn't needed. On top of that the majority of my CPU's are L5640's @3.7GHz which would probably stay cool enough if you blew on them. But in my farm (eee-iii-eee-iii-ooo) I plan to run some gulftowns at 4.2GHz which still doesn't push the 32nm Hexa's very hard. Even without fans I have a 970 in the loop and the temps are between 61-63c on the hotest core and I might be able to even turn down the Vcore a few notches, I really haven't tested the Vcore at all. L5640's are running 53-55c currently. So... I don't think keeping any of the individual CPU's cool will be hard.

I do have two future heat concerns when I start upping the number of computers in the loops though:

1) I overload the cooling potential of two car big car radiators. I should be OK, but 15 over clocked CPU's do add up, thats over 3,000W of heat being dumped into the loop by the cpu's and pumps and my ambient temperatures probably average 90F most of the year.
2) My fans don't force the hot air to exhaust out the window and draw fresh air into the garage then heat builds up inside the garage and ambient temperatures start to push to 110F or 120F or more.

I feel my design is good and both of these issues should be taken care of (knock on wood )

Heres my testloop in action (Fallwind and all the guys with the good cable management put me to shame )



And heres a Linux issue I ran into; I think I know why:


I'm guessing that 3GB of ram wasn't enough and the RAM was getting filled and causing all these Zero-status errors and screwing up everything. I just dropped a 4GB kit in this machine to test it out, if its running no problems in the morning then I found out the cause of the problem. I have two identical machines and linux installs running with 4GB and 6GB and zero problems.

[OldChap]

If I had built this I think I would have fitted the exhaust to the top of the window and the intake under the rads so that in summer the air is drawn from inside to outside by fan in spring/fall you might run fanless to some degree as heat rises and is exhausted naturally then in deep cold winter reverse the flow to heat the room some if outside temps are not extreme.

With the amount of rad you have My guess is that it would take closer to 50 processors to overpower this setup

Question....Which way is the prevailing wind relative to the window?

[Otis11]

If I had built this I think I would have fitted the exhaust to the top of the window and the intake under the rads so that in summer the air is drawn from inside to outside by fan in spring/fall you might run fanless to some degree as heat rises and is exhausted naturally then in deep cold winter reverse the flow to heat the room some if outside temps are not extreme.

With the amount of rad you have My guess is that it would take closer to 50 processors to overpower this setup

Question....Which way is the prevailing wind relative to the window?

Yeah, that deep cold winter they get in Florida... Brave sole he is.

But exhausting up wouldn't be a bad idea... don't think it matters enough to rebuild though.

Prevailing wind is a good question though... is it going to blow into this window and overpower your fans?

BTW - you running and GPUGrid on there trn?

[trn]

If I had built this I think I would have fitted the exhaust to the top of the window and the intake under the rads so that in summer the air is drawn from inside to outside by fan in spring/fall you might run fanless to some degree as heat rises and is exhausted naturally then in deep cold winter reverse the flow to heat the room some if outside temps are not extreme.

With the amount of rad you have My guess is that it would take closer to 50 processors to overpower this setup

Question....Which way is the prevailing wind relative to the window?

I would have rather build like this but i've built for my South Florida enviroment which means... summer 9 months of the year and fall/spring mashed into one non-summer period and 3 days of "winter." If anyone else is inspired to modify this type of design for their own project they should consider their environment. I don't get much wind, if any wind I would say parallel to the exterior exhaust window. Mounting my rads at 5' or more off the floor i've also run into a vertical space crunch with my massively tall reservoirs/manifolds, oops.

The reason I chose the top down build is because blowing rain can get in the window. With the fans and pumps mounted above the window they are safe from water that might short them out. I'm going to have to take my chances with the fans forcing enough air out the window; these sunon's I ordered look to have good static pressure and at 660cfm x 6 fans I roughly estimate they can turn over the air in my garage in 1.5 minutes.

Yeah, that deep cold winter they get in Florida... Brave sole he is.

But exhausting up wouldn't be a bad idea... don't think it matters enough to rebuild though.

Prevailing wind is a good question though... is it going to blow into this window and overpower your fans?

BTW - you running and GPUGrid on there trn?

I've been on GPUGrid for ~6 months. Right now my production has been down while I've been focusing on my rebuild. The gtx460 pictured I had to pull from the grid recently. That host is running Win XP x64 (ub34 l33t h4x0r3d supr3m3 version or somthing lame like that ) and the GPUGrid WU's were erroring out like crazy. I think its a problem with the OS and I need to reinstall a better copy of XP x64 (I have an oem copy that should work better I hope.) I also have a 2nd gtx460 that is off with MSI RMA service now. I bought it DOA out of the country ~ 2 months ago and finally got around to sending it in for US RMA (lets hope they can't figure out were I bought it; I pulled the Chinese stickers off the side, shhhh ) When I get my GPU's all back up and running I hope to have:

2 gtx460's running on Win XP x64 host
1 gtx280 on a win7 box

And future upgrades to my main rig will include Nvidia GPU's so they can game (with my 3 monitor setup) and Crunch. I like GPUGrid but they lack the professionalism of WCG to inspire me to outfit all my rigs with GPUs or build a crazy GPU crunching farm.

[onex]

trn, do you have anywhere where air can come INTO the garage?
cause otherwise you lower the air pressure while pushing air out, and to even that air would have to come in,
so you'r garage should look like a cooling case with push pull structure,
you can take what OC has said and set the exhaust exit on top of the intake entrance,
hot air would elevate up out side the widow and cooler air should come in from outside.

maybe you should set an air tunnel to move the hot air out of any cooler air reach, something like an L shaped wooden sleeve running out from the outtake fans,
wooden as it will not gather heat and maybe coating it with wax or something so it wouldn't get rotten when it rain,
so you should actually set it as an S with a bottom opening for the hot air to come out so water can't come in and fill you'r place in winter (if you have any).

110-120F is unacceptable, how can you even be there with such temps? and humidity would make you feel like you'r bathing in soup!

just be sure not to steam any of you'r neighbors!

[Rise]

I was thinking along the lines of onex about the negative pressure inside the garage but it's probably not a real issue. Air will come in through leaks all over the place, much like a manufacturing plant that tend to exhaust much more than they purposefully intake. If it is an issue, you could always install a vent in the garage (which it might already have anyway) and it will breathe fine.

trn - nice job designing the loop overall, I think you did a good job of trying to take everything into account that you could. What's your timeline for the other rigs?

[trn]

I was thinking along the lines of onex about the negative pressure inside the garage but it's probably not a real issue. Air will come in through leaks all over the place, much like a manufacturing plant that tend to exhaust much more than they purposefully intake. If it is an issue, you could always install a vent in the garage (which it might already have anyway) and it will breathe fine.

trn - nice job designing the loop overall, I think you did a good job of trying to take everything into account that you could. What's your timeline for the other rigs?

I've removed the weather stripping from the top edge of my garage door for exactly the reasons that OneX mentions. Its a two car garage door so 20 or so feet long and the gap ranges from about 1-3/4" to 1-1/4" over the width of the door. A breeze can blow through this gap so i'm thinking this should be enough intake for my fans but testing will tell.

Unfortunately i'm dead in the water waiting for parts from industry supply companies. I received my 'taco' pump for boosting flow from grainger in a day but the flanges are back-ordered so that doesn't do me any good... No idea when my fans will get here, I ordered from Newark and the fans will ship from the manufacturer warehouse directly I think Newarks supply was 5 fans and I ordered 6 So i'm stuck waiting there... and I don't expect quick service from Newark or their supplier so i'm stuck waiting for fans and mounting flanges

So i've been taking this opportunity to scour ebay and try and get some good deals on more gigabyte UD3R's, I picked up a Rev 1.0 last night for $165 (I hope it works with a 32nm Gulftown or Westmere because I just sent out my last 45nm 1366 this morning; i'm thinking I should be OK, my other Rev1.0's say works with 32nm on their box.) I need to start searching for more DDR3 also and i'm sure i'm still short some PSU's. I've lost count of what numbers of components I have and what I still need

[OldChap]

OldChap here ...and judging by the fact I did not remember you are in deepest coldest Florida I'm guessing I lost a few more brain cells...why is it the memory ones all the time?

Obviously my thoughts on the build were more in keeping with where I live. Here wind is a factor for 8 months of the year and the gales we often get have overpowered extract fans before here

[trn]

OldChap here ...and judging by the fact I did not remember you are in deepest coldest Florida I'm guessing I lost a few more brain cells...why is it the memory ones all the time?

Obviously my thoughts on the build were more in keeping with where I live. Here wind is a factor for 8 months of the year and the gales we often get have overpowered extract fans before here

I can see why (the wind, not the brain cells ) I just checked a map to see were Plymouth is. That could make for some good cooling situations. if you kept computers inside and ran the plumbing outside right to a huge car radiator sitting outside in some gale force winds and cool temps. Brass/copper rads are weather resistant anyways so just let that thing sit outside passively with pumps and computers inside...

Anyone can adapt these sorts of builds to best suit their environments. I want to see someone near a lake or river run submerged piping like a geo-thermal cooling, except this would be hydro-thermal... That would be a cool XS first build and much easier to sink piping in water than dig trenches for geothermal.

[trn]

My post have been rather photo-less while i've been waiting for parts so...

My "Taco" flanges shows up tonight out of the blue so now I can rebuild my loop this weekend with an extra pump to make sure that I have a nice level of overkill with my flow before I start building up the farm then have to tear everything down later to fix it I'm still waiting for my fans from Newark; it would really make my day if they just showed up tomorrow but that seems to stand a snowballs chance in hell...

My Green Taco with flanges




I bought 1-1/2" flanges for 1-1/2" ID Plumbing pipes; i'm thinking I can buy more plastic barbs at HD and they fit perfectly.

Salvaged a powercord from an old xerox laser printer that died at work I'll get 1 more rig up and running this weekend.


Plus 3 Linux hexa's

Boincview p0rn: My 6 windows hexa's


Hope my fans show up soon !

[Sparky]

I tell you I've been terrible about those power cords................. I feel really bad about it, but they are never open when I get off work

[trn]

I tell you I've been terrible about those power cords................. I feel really bad about it, but they are never open when I get off work

No, my bad. I'm still stalled without fans and I really appreciate the help

[Rise]

do you need anything else? I also have a crap ton of power cords, vga cables, maybe even a few dvi cables, red sata cables, etc that I'm never going to use.

[trn]

Thanks, I think the last things are major items More motherboards and ram and PSU's.