Build log: storage-oriented SR-2

[tiro_uspsss]

It's not an SR-2 any more, so maybe I should start a new thread...

nice mobo!

post a link to new thread in here please, when u do

[Anvil]

It's not an SR-2 any more, so maybe I should start a new thread...

I think you should.

Lets hope you don't have to look for yet another MB

[AceNZ]

Link to new thread:

http://www.xtremesystems.org/forums/....php?p=4474890

[AceNZ]

I'm attaching results from the 4W and 16W cached profiles from GullLars.

Both runs were on the same array: 32K strip, 8R0, DCE, NRA, DIO, WT, without the FP key.

[Computurd]

well, i guess if you could take the FP key off, and run the same profiles, we could get an accurate description of the latency differences.

[AceNZ]

well, i guess if you could take the FP key off, and run the same profiles, we could get an accurate description of the latency differences.

Those profiles *were* with the FP key off.

Unfortunately, I didn't have time to run the full suite of profiles before I had to replace the SR-2.

Or did you mean take the key off for tests on on the new motherboard?

[Computurd]

yes take the keys off for comparisons on the new motherboard. latency is latency, even though you will be 'crippling' the 9260 it will give us a apples-to-apples comparison. if we can get Gullars to throw up a comparative graph with quality of service (QOS) numbers i am extremely confident that it is there that you will see the tremendous difference. the latency is where you will see the improvement.

[AceNZ]

I've attached a zip with the results from the X8DTH, with and without FP.

Quick comparison of average response time, for 4W, no FP (QOS = IOPS / response time in ms):

QD1:
X8DTH: 0.1990 ms, 20023 IOPS, QOS = 100618
SR-2: 0.1998 ms, 19938 IOPS, QOS = 99789
--> QOS difference = 0.8%

QD32:
X8DTH: 1.4465 ms, 88458 IOPS, QOS = 61153
SR-2: 1.4490 ms, 88307 IOPS, QOS = 60943
--> QOS difference = 0.3%

[Anvil]

If there are no other deviances I'd keep the SR-2. (as it overclocks)

< 1% is nada, nothing, zilch. (imho)

The question is, what would a second CPU bring to the table?

[Sam_oslo]

I've been playing with SR-2 for a short time, and it is very sensitive to IOH-voltage. Too little of it messes up the PCIe dramatically and reduces the 3D-performance a lot, even tho it seams stable and can run Prime95. The QPI of 5520 chipset is designed to talk to tow CPUs, and using one CPU may have some kind of negative effect on PCIe too?.

Is it possible that PCIe-settings/performance has been affecting the performance here?

[AceNZ]

If there are no other deviances I'd keep the SR-2. (as it overclocks)

That's the plan.

The question is, what would a second CPU bring to the table?

I'm wondering about that, too.

I've been playing with SR-2 for a short time, and it is very sensitive to IOH-voltage. Too little of it messes up the PCIe dramatically and reduces the 3D-performance a lot, even tho it seams stable and can run Prime95. The QPI of 5520 chipset is designed to talk to tow CPUs, and using one CPU may have some kind of negative effect on PCIe too?.

Is it possible that PCIe-settings/performance has been affecting the performance here?

Definitely possible. I'll try bumping up the IOH voltage a little, to see if it makes a difference.

[AceNZ]

With 8 drives per controller, what's the best RAID configuration?

I'm finally starting to look at write performance--looks like WB might make sense after all, in spite of the huge drop in 4K random.

Ideally, I'd like to have some redundancy, but the performance tests I've done so far show pretty terrible results for 6+1R5. I'm considering 7R0 + 1 hot spare, to leverage LSI's SSD Guard technology.

I added another three 4GB DIMMs, up to 24GB total now. Seems to be working fine. CPU temps with Linx are 55C or so, with the CPU0 PWM at 85C, and ambient around 14C. I'll look around to see if I have an extra fan that isn't one of those high-pitched screamers -- might be useful for the PWM. Next step after that is a mild OC.

I also tried some testing with an HP SAS Expander. I was hoping to do some testing with 16 drives on one controller, but so far I can't get the damn thing to work.

[AceNZ]

A few preliminary PC Vantage HDD Suite results:

32K strip, NRA, DCD, DIO:

6+1R5, WT: 20129
8R0, WT: 27510
6+1R5, WB: 57490
8R0, WB: 67234

So, the scores more than double with WB enabled (almost triple with the R5 setup), in spite of the big decline in 4K random reads. I'm happy again now that I got the BBUs for the RAID cards.

I solved my overheating PWM problem by moving the fan I had blowing on the RAID cards on top of the CPU heatsink, blowing down toward the motherboard. It's a tight fit, and a bit of a hack, but it seems to work, based on a short Linx run. I'll do a full run soon to make sure (takes ages using the "All" setting with 24GB of RAM).

First, though, I noticed that the 14cm fan in the top of the PC-P80 is blowing into the case, instead of exhausting. In addition to being a pretty obvious thing, the description on their web site even says it should be an exhaust fan (hot air rises and all that)--so I'm going to try to flip it over to get it flowing in the right direction.

Temporarily, I put the factory fan back on the 5-drive HDD cage. It sucks air through the cage and exhausts it toward the RAID cards, but not with as much volume as I'd like, and it's also too damn loud. I'll have to find or buy a replacement.

[Anvil]

WB does make sense, for some reason I've found the Intels to handle WT worse than other drives on writes. I almost always end up using DCE using the Intels

Did you try raid-10 vs raid-5?
I haven't spent much time with raid-5 on the 9260, it might not be the best performer?

The pcmv scores seem to be on the low side.

2R0 X25-V on the 9260 scored > 53K. (not running OS though)

Speaking of BBUs, mine suddenly stopped working.
[Fatal] Battery needs to be replaced, SOH Bad.

I started a learn cycle but that didn't help, it completed the relearn cycle but as soon as it finished the messages popped up again.
It's been acting strange ever since I got it.

You'd better configure the Alerts just to be on the safe side.

[Computurd]

If i remember correctly, the intels do not have any on-board cache, which might explain the poor performance in WT?
tbh i have never tried R5 with my array, this intrigues me...i thought for sure R0 was king with speed...maybe time for a test?

[AceNZ]

Did you try raid-10 vs raid-5?

Yes. PC Vantage HDD Suite for with 32K strip, WB, DCE, DIO, FP:
4+4R10 = 53518
6+1R5 = 57910
8R0 = 68718

If i remember correctly, the intels do not have any on-board cache, which might explain the poor performance in WT?

I believe the Intels have a write cache. Setting DCE for the 9260 results in a slight improvement in PCMV HDD scores:

8R0 WB, DCD = 67234
8R0 WB, DCE = 68718

tbh i have never tried R5 with my array, this intrigues me...i thought for sure R0 was king with speed...maybe time for a test?

R5 / R50 isn't too bad for reads; it's writes that are much more costly--that's also where the controller's cache can be hugely helpful.

Here are a few CDM screenshots.

6+1R5 WT vs. WB:


8R0 WT vs. WB:


8R0 WT vs. WB in cache (100MB):

[Anvil]

mbreslin should do some R5 tests using the C300s or maybe someone with 5-6 SF based drives, I'm pretty sure the writes would improve vs the Intels.

The so called cache on the Intel is not used for caching user data.

BTW, when was the last time you cleaned the drives?

[AceNZ]

BTW, when was the last time you cleaned the drives?

Never; it's quite a hassle for 16 drives--so they're in steady state. I'm planning to re-cable and hook them up to the ICH10R and clean / secure erase before I do the "final" OS install. I'm also planning to underprovision to 68GiB per drive (vs. 74.5 GiB capacity = 80 GB).

[Anvil]

I expect the drives are somewhat degraded, some cleaning would be in place.

A quick format is all you need, using the ICH that is.
It does save quite some time compared to HDDerase.

In order to get more OP you do have to use HDDerase though.

[Sandon]

It doesn't surprise me raid5 gives better speeds than raid10. A lot of brands of controllers (LSI, 3ware, etc..) will NOT read from all the disks in the array (just half). Its retarded.

I know both Areca and Adaptec will read fro all the disks though.

There for assuming no other bottlenecks in a 8 disk array with raid10 you get only 4 disks worth of read/write bandwidth and with raid5 you get 7 disks worth of read/write bandwidth.

[Computurd]

the only problem with that is parity, there will be additional writes. My old barefoots are tired as it is

[AceNZ]

Here's a summary of PCMV HDD Suite results for various configurations.

They all have NRA and DIO, except as noted:

32K, 8R0, WT, DCD, 27510
32K, 8R0, WB, DCE, Cached I/O, 66345
32K, 8R0, WB, DCD, 67234
32K, 8R0, WB, DCE, 68178

32K, 6+1R5, WT, DCD, 20129
16K, 6+1R5, WB, DCE, 39056
64K, 6+1R5, WB, DCE, 51687
32K, 6+1R5, WB, DCD, 57490

32K, 4+4R10, WB, DCE, 53518
32K, 7+1R5, WB, DCE, 57721
32K, 7R0, WB, DCE, 58307
32K, 8R0*2S1 (16 drives), WB, DCE, 62192

16K, 2*(3+1)R50, WB, DCE, 57796
64K, 2*(3+1)R50, WB, DCE, 65928
32K, 2*(3+1)R50, WB, DCE, 71339

I also installed W7 x64 on the SSDs. First run results for the full suite were 20458 PCMarks.

With a 1.2M hour MTBF--the same as a typical midline HDD--it's hard for me to justify using an R0 config. With 400 to 500GB per volume, the time, hassle and expense of managing multiple backups per day is more than I want to deal with. I originally planned to give up half of the drives to mirroring; fortunately, it looks like a parity setup will work as well or better.

To my surprise, the 2*(3+1)R50 config with a 32K strip is the fastest. True, it increases the number of writes--but OTOH, those extra writes are spread out over 8 drives. Given how fast the technology is changing, the speed improvement seems worth the risk.

[AceNZ]

Well, a bad day on the cooling front...

I solved my overheating PWM problem by moving the fan I had blowing on the RAID cards on top of the CPU heatsink, blowing down toward the motherboard. It's a tight fit, and a bit of a hack, but it seems to work, based on a short Linx run. I'll do a full run soon to make sure (takes ages using the "All" setting with 24GB of RAM).

Well, this didn't work after all. The CPU PWM temp as reported by E-LEET still jumps to 85C or higher (98C with a 166 BCLK and 1.3V Vcore). I tried using a smaller 40mm high-rpm fan so I could get it closer to the PWM heatsink, but still no joy. However, the reported temp jumps around so much, and returns to normal readings so quickly after the load is removed (within 4 or 5 seconds), that I'm beginning to suspect a faulty sensor.

First, though, I noticed that the 14cm fan in the top of the PC-P80 is blowing into the case, instead of exhausting. In addition to being a pretty obvious thing, the description on their web site even says it should be an exhaust fan (hot air rises and all that)--so I'm going to try to flip it over to get it flowing in the right direction.

On closer examination, the fan was pointed in the right direction after all. However, for some reason the air flows back into the case around parts of the outside edge--it seems to be bouncing off of the vent on the top of the case. Very odd and inefficient. I'd like to remove the vent entirely, but it's going to take some work with a dremel since the power switch and USB ports are mounted to the front part of it...

Temporarily, I put the factory fan back on the 5-drive HDD cage. It sucks air through the cage and exhausts it toward the RAID cards, but not with as much volume as I'd like, and it's also too damn loud. I'll have to find or buy a replacement.

This one was also a bust. A while after making the change, the RAID manager software reported a fault with two of the 9260 cards. Although it wasn't explicit (no message in the log, just a warning on the dashboard), without the better fan, the battery temperature was too high (125F).

I'm tempted to replace both the rear and top fans on the PC-P80. Either that, or bite the bullet and switch to water now that there are some good blocks out for the SR-2. I've never done water before, but it can't be too difficult, right?

[Computurd]

have you thought of removing the PWM heatsink and re-tim? this is a common problem with many motherboards, almost as a rule many will strip off any factory heatsinks on a board and re-tim them before using any board. i personally do as well, for some reason they seem to use some crappy tim, or have poor contact, or both. another issue might be warping (slight) of your board, which might cause some contact issues. be careful though! you wont want to much torque.

water is cool, i use full board blocks now personally. Since that is an overclocking board i would recommend it. I was walked through my first MB installation by a friend over videochat, which helps, there are many things to learn. WARNING: water can get expensive, fast!
some do question the necessity of motherboard blocks of course, but with the issues your having it will probably be warranted. you might try the re-tim first though.

how well is the air flowing into the case? if it isnt flowing in very well it of course will not flow out very well.

[xeon_1]

have you thought of removing the PWM heatsink and re-tim? this is a common problem with many motherboards, almost as a rule many will strip off any factory heatsinks on a board and re-tim them before using any board. i personally do as well, for some reason they seem to use some crappy tim, or have poor contact, or both. another issue might be warping (slight) of your board, which might cause some contact issues. be careful though! you wont want to much torque.

water is cool, i use full board blocks now personally. Since that is an overclocking board i would recommend it. I was walked through my first MB installation by a friend over videochat, which helps, there are many things to learn. WARNING: water can get expensive, fast!
some do question the necessity of motherboard blocks of course, but with the issues your having it will probably be warranted. you might try the re-tim first though.

how well is the air flowing into the case? if it isnt flowing in very well it of course will not flow out very well.

Yea on my classified they used some kind of ruber/elastique as tim.

And water get realy expensive very fast is was over 800€ to completely cool my rig.
But my vreg went down from 85°C to 45°C.
The hottest component in my rig currently is the raid card witch i didn't include in the cooling loop since people said it would be overkill.
Been thinking of slapping on of my old zalman vga coolers on to it the one form my 9700pro.
Or mayby one of the old stock cooler from my vga cards that i have in the house most are much larger then the standard heatsink of the arc-1222
It runs arround 50°C now.

But i would defenatly redo the tim of the vreg's