Build log: storage-oriented SR-2

[AceNZ]

I hadn't thought about re-seating the heatsink. Makes great sense, though. Unfortunately, after my recent CPU swaps with the X8DTH board, I'm temporarily out of TIM, so I'll have a wait until my next order arrives before giving it a try (one disadvantage of living in a small town is the lack of supplies for this hobby; nearly everything I get has to be shipped).

The airflow into the case is just OK, not great, in spite of the three 140mm fans in the door on the front--the full drive bays block much of the flow. However, even with the side cover removed, the flow out the rear and top fans is still poor. If I stay with air, I'll probably replace those two fans, and may have to get out the Dremel and a drill to really fix it.

I've started to look into water; no decisions yet, but it seems like a big step for this rig, given how full the case already is with the SR-2 and the drives. I'd probably have to mount the radiator on the top of the case somehow.

I did add two more 1TB RE3 drives to the system--so 8 of them total now, all on the 9261-8i. The last two drives are configured in R1; I'm planning to use them for write-heavy applications, such as the database log file.

[AceNZ]

Are there any voltages in particular that tend to be prime causes of high VREG temps? Are Vcore and VTT equal factors?

[Sam_oslo]

Are there any voltages in particular that tend to be prime causes of high VREG temps? Are Vcore and VTT equal factors?

Most av all vCore, but VVT too. A moderate VVT (under 1.34v) won't add much heat tho. You need active cooling on CPU#0 VREG to keep it in shock, anyways.

I'm using vCore=1.16v, IHO= 1.35v, VVT=1.33v for BCLK200x19, with a small (but with good airflow) fan over the CPU#0 VREG. It keeps under ~65'C under Prime-load. inside a closed case.

[Computurd]

umm also the PWM frequency setting in your bios on those evga will allow you to cook eggs if you set it to the highest setting

[AceNZ]

It turns out that I was somehow running the 4.27.0.64 version of the LSI driver, instead of 4.27.1.64.

Here are CDM results after upgrading:



Sequential read results right before this were 1177 MBps.

Still not quite the 2GBps+ that @CT and others are getting, but much better! Oddly, though, PCMV results are about the same.

The downside is that now I feel like an idiot for not noticing this before.

[Computurd]

rofl! I hate that. I was running an intel driver for their re-brand of the LSI and couldnt get fastpath to work for a few hours...was getting pissed off too!
gotta love those "AHA!" moments though.

quick question, are you able to run the new firmware for the 9260 on your motherboard? i know almost all varieties of classifieds are having probs, and jacob over at evga is working on it, but i am curious about your board....

(also i am still unconvinced that you arent having some negative impact from the NF200, even though you results state otherwise. maybe ive just had such a bad experience with the classified with teh NF200 that i am a NF200 hater forever...did you ever run the gullars profile on both boards though, so we can compare access times? sorry if i missed it...but with FP enabled i am very curious of results. i think we compared pre-FP and FP results...there are just so many others whose results have exactly mirrored mine...)

[AceNZ]

I haven't tried the latest LSI firmware yet -- too many others have reported problems with EVGA boards, that I thought I'd wait.

I did post gullars profile results for both the SR-2 and the X8DTH, but they were only for no-FP. I've also measured latency with Everest on both: the X8DTH was 0.06ms, and the SR-2 is 0.04 to 0.05ms. Even so, I agree there may be some interference from the NF200 -- but it wasn't enough to give up the ability to OC the Xeon (which is giving me headaches at the moment, too; "it's always something").

[Computurd]

yea being able to oc is the trump card for sure! Also, it seems that the implementation of the NF200 is somewhat better on that board, which is also why i was curious of your firmware status, maybe there is a better option rom on that board as well. I am sure it will work they have to have tons of option-rom for all them slots! i am happy you are getting good results!
What issues are you having with the OC?

[AceNZ]

I'm seeing three problems:

1. Devices sometimes stop responding, such as the keyboard or the RAID controllers; the USB mouse pointer still moves around when this happens, so I know the CPU is alive.
2. The first time I power-on, the system won't boot; I have to hit the reset switch after a minute or so. I've been wondering about a power dip due to all 8 HDDs coming on at the same time. I'm looking for a staggered power-up option somewhere in the LSI software....
3. The CPU0 PWM temperature can hit 73C when running PCMark Vantage. With small FFTs on Prime95, it reports 100C, but then drops to 56C within 5 seconds of stopping the app. I'm still not sure I believe the sensor, but I'm planning to re-seat the VREG heatsink anyway when some new TIM arrives (could take a few weeks -- looks like I'll have to completely remove the motherboard from the case to do that, though)

OC settings:
BCLK 166
RAM 1066 (8x)
With Vdroop
Vcore 1.28
VTT 1.25
IOH 1.25
VDIMM 1.55 ("Auto" put it at 1.68V according to E-LEET, which seemed high for 1.5V DIMMs that aren't being OC'd)
PCIe 100

I've tried tweaking Vcore and IOH up-and-down. Whichever way I go seems to make things worse, not better.

Ideas?

[Sam_oslo]

^^You have an optional floppy-connector right by CPU#1, that will feed juice to USB. Connecting that may helps. If you have problems with PCI-devices, you can try to connect the optional 6-pin connector right over PCI-lines too. This MB is so huge that it can't feed enough juice everywhere, I guess.

Higher IOH-voltage should have a big impact too, you can try IOH=1.35v.
You could also try to lower the "IOH QPI 0 Signal" to -85 too (it is under "signal tweaks"-menu).

SR-2 over-vols the Vdimm, I'm sing 1.61v (BIOS set) to get 1.65v in w7.

That PWM=100'C is scary stuff, it can kill the chip. I wouldn't dare to try it at home. I would stay away from Prime-runs until you fix the cooling.

[Computurd]

ummm many ideas... i would try and run my ram as loose as possible of course, at first. other than that i am not sure what is happening. definitely need to re-tim, but that is a waiting game. do you have a multimeter to check what your vdimm is really running at?
i also would definitely go without vdroop.

[AceNZ]

Thanks for the ideas. I've went back to stock settings, and still had a problem with the system hanging. Turns out that one of the 9260 cards was overheating, as reported by MSM. Guess three cards in three back-to-back slots is too much after all, even with the extra fan.

@Sam: I have all of the power connectors on the SR-2 hooked up, including those for CPU1 and the one at the top of the PCIe slots.

@CT: Yes, I have an old digital meter around somewhere. Would be interesting to compare to the readings in E-LEET. Looks like I should get some thermal pads, too, in addition to the TIM -- apparently the MOSFETs on some boards aren't completely even, which can impair the fit of the heatsink.

[Computurd]

yea and word of advice with thermal pads...when you buy some get extra, sometimes they are hard to find, i had to wait forever once for them to come back into stock!

[AceNZ]

I switched to using this box as my day-to-day PC a couple of days ago. I have to say, it has been quite a joy. I've installed all apps on a different volume / controller from the OS. That way, Windows will issue parallel reads from both drives for any required DLLs during app startup. Visual Studio 2010 Premium opens in about 2 seconds, compared to 20 seconds on my old PC. IE, most Office apps, Adobe Reader, etc, open pretty much instantly, even with huge files. Whenever the system feels a little slow, it's either because of the network or the HDDs -- as long as I stay on the SSDs, it feels blazing. Sometimes, file copies happen so fast I do a double-take to make sure they actually happened.

Over the last 4 or 5 days, I have had the system crash once with an LSI firmware error, and I've had one reboot that failed because the LSI firmware was in a "fault state." Not sure if it's a problem with one of the boards or possibly a thermal or firmware issue. I do know that with the BBUs, the boards stay pretty warm, even with the fan I have blowing on them -- MSM regularly reports 90F for the BBUs, with ambient at around 66F.

I'm dragging my feet on trying to fix the VREG heatsink problem, since with so much cabling, it's such a PITA to pull the motherboard out. So, still no OC. Probably just as well while I'm ironing out the remaining kinks.

I'm tempted to install the new FW for the 9260 to see if it works with the SR-2, but I have real data on my drives at this stage, so I'll probably wait until I get a couple of external drives setup for backups, unless the crashes become more frequent.

[Movieman]

^^You have an optional floppy-connector right by CPU#1, that will feed juice to USB. Connecting that may helps. If you have problems with PCI-devices, you can try to connect the optional 6-pin connector right over PCI-lines too. This MB is so huge that it can't feed enough juice everywhere, I guess.

Higher IOH-voltage should have a big impact too, you can try IOH=1.35v.
You could also try to lower the "IOH QPI 0 Signal" to -85 too (it is under "signal tweaks"-menu).

SR-2 over-vols the Vdimm, I'm sing 1.61v (BIOS set) to get 1.65v in w7.

That PWM=100'C is scary stuff, it can kill the chip. I wouldn't dare to try it at home. I would stay away from Prime-runs until you fix the cooling.

AceNZ:
Listen to Sam..almost..
I wouldn't touch the IOH QPI0 signal at all.
-70 is used with xtreme cooling or with super high mhz but your not doing that.
The issue with that PWM at 100C would scare the crap out of me though.
Get a fan on it and don't push the machine until you do.
Sam's also right on the board overvolting the memory.
setting for 1.65v and net you 1.69v..
Unless your shooting for super high memory speeds I'd setfor 1.57-1.58v in bios and that should get you 1.62-1.63 and thats more than enough.
Good Luck.

[AceNZ]

Thanks for the advice on voltage settings. Since I'm using ECC RAM, I'm thinking of staying at or very close to 1333, so hopefully they won't need much of an overvoltage, if any.

I put an 80mm Thermaltake fan on the VREG heatsink. It helps a little, but doesn't solve the problem. As Sam suggested, I'm staying away from prime runs until I find a better fix.

In addition to re-seating the heatsink or using water, I'm also thinking about air flow into the case. Turns out that the 2.5-inch bays are pretty effective at blocking airflow from the 140mm fans in the front. I may end up putting a few of the small 40mm fans back on the bays to pull some air through, although they would have to be with a resistor on the power cable to keep the noise reasonable. Also, both the 120mm exhaust fan on the rear of the case and the 140mm one on the top don't seem to move much air, even with the case wide open.

For the short term, I put the SuperMicro 92mm fan back on the HDD bay with a resistor. I'm sure it helps the HDDs, but unfortunately the RAID card temps increased... Sigh.

Another idea I had was to move the CPU from the CPU0 socket to the CPU1 socket. I'm only running a single CPU, and I heard that it should work in either socket. That approach might solve my CPU0 PWM temp problem, and it would also move the CPU close enough to the 120mm fan outlet in the back panel that I should be able to use a Corsair H50 heatsink -- which might be more effective than my current Noctua NH-U9DX.

[Anvil]

So I finally got the FP key

These are the results from 6R0 Intel, single 9260-8i, GB UD7.
(latest firmware and driver)

SQLIO 4K random read, 1W
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 152804.19
MBs/sec: 596.89
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 1
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 93 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0