Build Log: 100,000 PPD by 1/23/2011

[Rembulan]

@ D A Sorry Sir the rendering farm already full loaded may be after I can buy some SR I can spare some X58 with I920 for cruching

@Fallwind Thanks for Pic really open my mind

[hedge]

I remember that build. I think the thread was titled, "new barn for the farm animals" or something like that.

I agree with fallwind, the fans don't need to be right on the radiator, in fact it is probably better if it isn't. If the fan is right on the radiator, there is a dead spot in the middle of the fan where the air hardly moves. That is why fan shrouds were great.

[trn]

I agree with fallwind, the fans don't need to be right on the radiator, in fact it is probably better if it isn't. If the fan is right on the radiator, there is a dead spot in the middle of the fan where the air hardly moves. That is why fan shrouds were great.

I agree also, I built my radiator box with 2 x 8's so the interior depth is nearly 8 inches and my rads are only ~2" in height. The box is a shroud so the fans can slowly force air through the entire surface of the radiators. The fans i'm mounting on the top exterior surface of the Rad's.

I like the suggestion of using disassembled 115 off the shelf fans but for my project to work airflow is just to critical. If the fans are unable to force the air though my shroud, rads, then exhaust out the side window then the heated air will back up into my garage very quickly pushing interior temperatures over 100F. I drew a quickie diagram to illustrate exactly whats going on inside that massive black box that sits in front of the window:


So with airflow being so critical I went ahead and ordered 6 of these http://www.newark.com/sunon/a1259-mb...vac/dp/20M3858. Unfortunately it will probably take two weeks or so before these things show up, i'm not expecting them anytime soon. I feel the high static pressure of these monster fans ought to ensure hot air is exhausted out my window and cooler air is drawn into the garage. My air intake is along the top edge of my 2 car garage door, roughly 1" - 1.5" gap extending the length of the door.


For pump testing I installed 1 sub loop of 3 rigs. The sub loop has a 970 > L5640 > L5640 > Res. Right now I only have the first rig, the 970 up for testing. Disregard the voltages in the CPUz shot of the 970, I did not do any tweaking of the voltages or testing. I entered bios once, turned bclk up to 168 (x25) for 4.2GHz and then just took a guess at a nice safe voltage a gulftown should run 4.2GHz at. I have no idea what the minimal voltage is required to run a 970 at 4.2GHz is right now, thats an issue for later. Loop works OK and flow is so-so. Not bad temps for passive cooling To help make sure I have no future flow problems I went ahead and ordered one of these http://www.grainger.com/Grainger/TAC...C90?Pid=search along with flanges and O-Rings. I plan to install the pump between the 2 radiators (because there is lots of room for the pump there )



The Fc1 Gigabyte UD3R rev 2.0 bios doesn't seem to like bclk 205 like my rev 1.0 does. I don't want to push qpi voltages higher than 1.32v to try and stabalize 3.7GHz so I think i'll drop the rev 2.0's down to 3.6GHz.

And thats about it for now, I'm still working with Linux trying to figure out how to make an install work as a headless cruncher. I tried Ubuntu but am giving up on Ubuntu, I don't like Debian based distro's. I dislike how the config files are setup and by default too much crap is installed and enabled so i'm back to Archlinux. Linux todo:
- Pick a desktop enviroment that is suitable for a cruncher and GUI login. I installed KDE base on Arch... I dunno why I guess because the new KDE is sweet and I wanted to screw around with it but its not very suitable for a cruncher; I think i'll probably end up with Xfce4. Fluxbox is OK but even more minimal than I need.
- Make x11vnc startup with a GUI login everytime the computer boots. I installed KDM but had no luck making x11vnc automatically yet. I'm sure this is jut a matter of screwing around with the setup more. Plus i think i'll install xdm along with xfce4 and ditch KDE and KDM.
- Boinc starts as a daemon... but doesn't do anything, it crashes or something . The process is there, but does no work and the manager can't connect to it. Its not until I kill the process and restart boinc in a tty under my username that it starts to do work and I can connect with the manager. The daemon starts boinc with the user "boinc" which I thought might have been the culprit but I can't figure this one out yet, on my ubuntu machine the daemon starts up under the boinc user and works
- Conkey; I need a task manager since xfce4 doesn't have a good task manager. Conkey seems OK... I don't like that it merges all the cores into one cpu usage bar (though maybe this is configurable??) And conkey shows a basic temperature (although I think for only 1 cpu??, but again I dunno.) So anyways I have to mess around with this and get a good conkey config.
- Figure out how to make the hostname of the Linux machines discoverable to Windows. My ubuntu install does this by default so I can connect to linux1.local (ubuntu machine) with VNC or whatnot. I can't connect to arch1.local or arch1 or anything, I have to manually find and connect to the IP which sucks. I think this has to do with IvP6 settings or somthing, again

So still a lot of work to be done Linux wise, but I have downtime now while waiting for fans and extra pump.

[OldChap]

How are the boards insulated from the racking? I ran some with neoprene under for a while and it all got stuck to the hot parts of the board. (sorry if you posted this already and I missed it)

[D_A]

That's why I run Ubuntu Server for my headless machines. VERY minimal install, no need for auto login since I don't use a desktop environment on them at all (I use ssh). I also use static IPs for machines on my network.

[trn]

How are the boards insulated from the racking? I ran some with neoprene under for a while and it all got stuck to the hot parts of the board. (sorry if you posted this already and I missed it)

Good question, my first plan was to buy sheets of window glass to insulate the boards. A sheet of window glass comes precut and is like $2 or something very cheap. But right now i'm not sure if its needed, I bought metal wire shelves and they have a black epoxy coating on them, so while testing I just zip tie'd the boards directly to the shelves. So far zero heat problems or electrical grounding. While I continue to run my 1 test loop i'll keep a eye on this, but if it continues not to be an issue i'll just zip tie em right to the shelving.

[OldChap]

I guess I was thinking that running 24/7 could just about produce enough heat on a component to "melt" the coating on the wire and give problems to say the least. I like the idea of glass by the way... how about you get say 3 * 2 inch strips per board...enough to insulate but lets the air get to the back of the board still. Just a thought

[trn]

That's why I run Ubuntu Server for my headless machines. VERY minimal install, no need for auto login since I don't use a desktop environment on them at all (I use ssh). I also use static IPs for machines on my network.

My ubuntu install is unbuntu server, but I did install the desktop environment which then makes it pretty much the same as normal ubuntu .) Ubuntu server is only minimal compared to unbuntu desktop Even without the desktop enviroment it was asking me for upgrades and stuff like that. I'm sure its not bad, i'm just not liking these debian based distro's. I like files and folders to be more BSD like. If boinc ran native on BSD without Linux emulation I would be running FreeBSD on my *nix crunchers. Hell, I still might even give FreeBSD a go, I hear the linux emulation no or almost no effect on boinc performance.

I use ssh also for most stuff but need a desktop enviroment for 10% of the admining (boincmgr really is the only thing.) Boinccmd is just awful and since the only thing these computers will be doing is running Boinc it would be nice to have Boincmgr.

I haven't run *nix in years, i'm finding it to be a bit like riding a bike, the basics came right back to me but I sure am rusty as hell! Btw, I have been impressed with the growth of Linux and the Distro's! Many of these newer distro's do all the hardware setup for you. LiveCD's are impressive, stuff just works with GUI's and the likes. Package managers have gui's and do most of the configuration of newly installed programs for you. KDE 4.3 is really a cool and configurable WDM. Linux has progressed to the point that an average person could install and run Linux as easily as Windows. If it wasn't for programs and games not being available on Linux I would run it on my main rig no questions.

[trn]

I guess I was thinking that running 24/7 could just about produce enough heat on a component to "melt" the coating on the wire and give problems to say the least. I like the idea of glass by the way... how about you get say 3 * 2 inch strips per board...enough to insulate but lets the air get to the back of the board still. Just a thought

A few days of running a 970 at 4.2GHz in 90F+ ambient temperatures ought to tell me if the coating is OK or not. I've been running the 970 all day; so far the backplate is warm but not too hot to touch.

[D_A]

Most distros now give you the option of automatic updates (you can turn it off in Ubuntu Server during the install if you prefer) and you can use the BOINC manager on your main machine to access and control BOINC on a headless one, there's no need to install the package on anything but your main rig.

[Sparky]

Good question, my first plan was to buy sheets of window glass to insulate the boards. A sheet of window glass comes precut and is like $2 or something very cheap. But right now i'm not sure if its needed, I bought metal wire shelves and they have a black epoxy coating on them, so while testing I just zip tie'd the boards directly to the shelves. So far zero heat problems or electrical grounding. While I continue to run my 1 test loop i'll keep a eye on this, but if it continues not to be an issue i'll just zip tie em right to the shelving.

Ehh I've seen that coating crack too many times to want to do that. Plus, sharp solder points could scratch and boom.

Honestly, I'd take a slab of cardboard to stick between the board and shelf. Free and works great. Ask me how I know...

[trn]

Most distros now give you the option of automatic updates (you can turn it off in Ubuntu Server during the install if you prefer) and you can use the BOINC manager on your main machine to access and control BOINC on a headless one, there's no need to install the package on anything but your main rig.

Oh yeah... Duuuuh! Thanks for the great idea; instead of screwing around with WM's and WDM's and vnc i'll just do it the easy way Now I just have to fix my boinc daemon and i'm good to do with linux

Ehh I've seen that coating crack too many times to want to do that. Plus, sharp solder points could scratch and boom.

Ok, that sold me i'll find a way to insulate the boards

[D_A]

Oh yeah... Duuuuh! Thanks for the great idea; instead of screwing around with WM's and WDM's and vnc i'll just do it the easy way Now I just have to fix my boinc daemon and i'm good to do with linux



Ok, that sold me i'll find a way to insulate the boards

Yeah, if you've played with the command line before it really is quite simple.

[Biker]

I have 3 X58A-UD3's rev 2.0 running at the moment and in my experience I estimate they are all undervolting the QPI by .05-.07v so if you set 1.35v in the bios then the real voltage is only 1.28-1.29v.

This is the same on the FA and FB bios...

I do not know the QPI read points (feel free to point them out if anyone knows the location ) so I cannot confirm but the chips run stable at 1.275 QPI on other boards that I have measured with a DMM and they require 1.35 set in bios minimum for stability at the same clocks on the Gigabytes...

[trn]

I have 3 X58A-UD3's rev 2.0 running at the moment and in my experience I estimate they are all undervolting the QPI by .05-.07v so if you set 1.35v in the bios then the real voltage is only 1.28-1.29v.

This is the same on the FA and FB bios...

I do not know the QPI read points (feel free to point them out if anyone knows the location ) so I cannot confirm but the chips run stable at 1.275 QPI on other boards that I have measured with a DMM and they require 1.35 set in bios minimum for stability at the same clocks on the Gigabytes...

Good info; I use the Level 1 Line calibration which undervolts, but maybe not as bad as stock. I'm assuming QPI is undervolting more than the bios reading. I was currently trying bclk 205 at 1.32v QPI which works on my rev 1.0's, I'll try a bump to 1.34 and see if that works.

[Snow Crash]

Perhaps I am over my head with this crowd but ... I've found that reducing the memory multi makes no difference on the crunching runtimes but it does allow me to lower the QPI freq (cuz minimum QPI is 2x ram) which in turn lets me run lower QPI volts.

[jcool]

Perhaps I am over my head with this crowd but ... I've found that reducing the memory multi makes no difference on the crunching runtimes but it does allow me to lower the QPI freq (cuz minimum QPI is 2x ram) which in turn lets me run lower QPI volts.

You mean Uncore frequency?
QPI frequency is BCLK * 36x/40x/44x (also known as 4,8/5,6/6,4 GT/s).

Min. uncore speed = Dram speed * 2 is only valid on 45nm Bloomfields, by the way. Gulftown/Westmere are operating on a minimum uncore speed of Dram * 1,5

On another note, I really hate how Asus, Gigabyte and most others are confusing the user with their stupid bios naming (only DFI and EVGA do it right AFAIK). What they call "QPI voltage" in bios is, in fact, CPU VTT voltage, same as with good old LGA775. It has absolutely nothing to do with the QPI link

They might as well have called it "BCLK voltage" so ppl would know what to crank up for high BCLKs

Edit: Almost forgot... nice progress there trn, keep it coming
And yeah, I wouldn't recommend going past 1,3V VTT on a 32nm A0 for 24/7 crunching, so considering the Vdroop, 1,35V Bios VTT is the max I would recommend.
Also try and keep the uncore (CPU-Z reports it as "NB Speed") at or below 3Ghz, as that will lower your VTT requirements immensely.

[onex]

Ok, that sold me i'll find a way to insulate the boards

you should try maybe plywood?
these 5mm thick pieces, isolates, flexible, cut-able, non fragile and soft on taking any sharp edges .

[trn]

you should try maybe plywood?
these 5mm thick pieces, isolates, flexible, cut-able, non fragile and soft on taking any sharp edges .

Which is a good idea because I have lots of scrap plywood available and can cut some strips to stand the boards off the shelves and still allow good airflow. seems low tech but ought to work nicely.

[onex]

btw,

On another note, I really hate how Asus, Gigabyte and most others are confusing the user with their stupid bios naming (only DFI and EVGA do it right AFAIK). What they call "QPI voltage" in bios is, in fact, CPU VTT voltage, same as with good old LGA775. It has absolutely nothing to do with the QPI link

VTT is (was) the FSB termination voltage so that can be that ASUS and GB call it so as FSB was replaced by QPI.

[trn]

Naming scheme or not; I do prefer how GB handles VTT/QPI voltage in bios vs. EVGA. Setting 1.32v (gb) or whatever setting is much more logical than +200mv (evga)

[jcool]

Naming scheme or not; I do prefer how GB handles VTT/QPI voltage in bios vs. EVGA. Setting 1.32v (gb) or whatever setting is much more logical than +200mv (evga)

Yeah I didn't like that on the E760. Thankfully evga has already changed it, at least on my SR-2 you can directly type in the desired VTT voltage

What I never understood with GB is why none of their boards have VTT readout. I even asked one of their reps at CeBit, and he said "use Easytune".
Guess what, Easytune doesn't show it either
Besides I could never change even 1 voltage setting with Easytune on my GB Extreme without causing a hardlock
Does it work on the UD3R?

[Biker]

Easytune???

I'm not rushing back to hardlock city either


trn, this article confirms the VTT / QPI undervolt on the X58A-UD7 rev 2 and since the UD3 rev 2 is close to the same board I think this is more supporting evidence to back up the same undervolt happening on the our boards...

I am checking the read points now to confirm but I am far from 100% they are in the same spot as shown on in the link above for the UD7 rev 2...

[trn]

Biker, Good read. I went ahead and upped the QPI(vtt ) voltage to 1.34 on my UD3R rev 2.0's running at bclk 205 and so far they have been stable as a rock.

Easytune sucks!!!! I abandoned it long ago and will never go back! ET is good for nothing but hardlocks and it doesn't even seem to read the voltages right (before it hardlocks.) And yeah, i'm using an E760 on my main rig so I get to deal with the wacky +mv VTT Can't say it matters much these days since I have a 980x in it for now and never mess with the 980x really or even have time to tweak settings etc on this rig; much more work to be done elsewhere

I found one L5640 in my batch that seems to not like to OC last night. I'll mess around with it later to see exactly what that one problem chip can do, not sure if I know how to run a CPU with stock settings

And I finished setting up my test loop, I have a 970 @ 4.2GHz, and 2 L5640's at 3.7GHz on one loop and an air cooled gtx460 to 'stress test' my cooling (no fans still.) So far passively the loop is handling it just fine, temps on the 970 only raised 1 - 2c when the extra computers and 460's were added; looks like a good sign once I get the system rebuilt with an extra pump and have fans

And despite all my previous ing and whining about ubuntu I went ahead and installed ubuntu server on some more computers and for now plan on using ubuntu server for more crunching rigs. I still don't like the debian file system, I still don't like apt-get, I don't like sudo vs su, I don't like the mess of package systems deb apt-get apptitude and whatever else its got, and the ubuntu wiki isn't that great. But it, ubuntu works

[D_A]

You can still use su if you want to. The Ubuntu forums gods will try to rain thunder, torment and bannage on me for mentioning this (blow me, Ubuntu mods), as it's not "the Ubuntu way", but you can type "sudo su" and you're there. No need for root password.
You also don't HAVE to install everything from the repos, either. Compiling things yourself works just fine and you can either do it "old skool" and leave the fresh toys in whatever directory you like or compile into deb packages and use the package manager.