Build log: storage-oriented SR-2

[Computurd]

all of your slots except slot 1 will go through NF if it follows the pattern they used with the classified.
i will tell you this....i had to get rid of my original classified because of the latency that was introduced into my system by the NF200...i had to go with the non-nf version of my motherboard because the NF will limit your maximum IOPS, plus add a latency hit that is unbelievably bad. Dude, the max i could get out of my array with the NF board was 1330 mb/s (for instance) yet it is over 1700 without. any raid card on that nf is going to be so strangled it is utterly ridiculous. on top of that i had tri-sli, and when enabled you could forget it, the latency and IOPS got even worse. I cannot stress to you or impress upon you enough the amount of pure power that is going to be robbed from you via that NF200 chip. I have 500 bucks right now that says within six months they release a refresh without the NF, just like they did with the classified, and it is BS bro. they just include it because of Nvidias arm twisting to my understanding. "put the chips on there or you dont get SLI on your motherboards" is basically what i have heard.
not trying to be overdramatic here, but seriously it suxors

[xeon_1]

Didn't the non nf200 have a max config of 16x16x 8x8x or something where as the nf200 version has 16x16x16x8x

[AceNZ]

all of your slots except slot 1 will go through NF if it follows the pattern they used with the classified.

The SR-2 has dual NF200's; in order to support quad-sli, all of the slots go through them. With the tri-sli boards, two x16's go through the NF200 and one goes direct to the IOH.

i will tell you this....i had to get rid of my original classified because of the latency that was introduced into my system by the NF200...i had to go with the non-nf version of my motherboard because the NF will limit your maximum IOPS, plus add a latency hit that is unbelievably bad.

Unfortunately, I'm starting to come to same conclusion.

they just include it because of Nvidias arm twisting to my understanding. "put the chips on there or you dont get SLI on your motherboards" is basically what i have heard.
not trying to be overdramatic here, but seriously it suxors

I didn't find out about the dual-NF200 setup on the SR-2 until after I placed my order. I knew it had a 5520, and with dual-Xeons and all those slots, just couldn't imagine that it would be a single-5520 with dual-NF200's, instead of dual 5520's. If I'd known about the NF200's, I don't think I would have gone this route.

I'm wondering now if it might be possible to overcome some of the NF200 latency by overclocking. I'm doubtful, but it seems worth a try before swapping it out for a server board.

BTW, several people have asked what @Tilt has done to make his system so fast. I'm increasingly convinced that the answer is easy: he's running a server board, with real I/O capability, instead of these hacked sli-focused boards (which includes things like the Asus "Supercomputer", etc).

Didn't the non nf200 have a max config of 16x16x 8x8x or something where as the nf200 version has 16x16x16x8x

Non-NF200 boards with 5520's or X58's are limited to 36 PCIe lanes; often 32 of them are exposed in slots -- so maybe x16x8x8, or x16x8x4x4, or something like that, usually with some kind of switching thrown in for either/or type configs.

[Xtrafresh]

This thread makes for a riveting read! Awesome build, and really informative too.

Didn't the non nf200 have a max config of 16x16x 8x8x or something where as the nf200 version has 16x16x16x8x

Well, since the X58 only had 36 lanes, that would be x16-x8-x8 for tri, and x8-x8-x8-x8 for quad, with four lanes going to the SB.

I'm wondering now it might be possible to overcome some of the NF200 latency by overclocking. I'm doubtful, but it seems worth a try before swapping it out for a server board.

I'm confused, can the NF200 be OCed? Or is there something else that can reduce it's latency?

[Computurd]

you can bump the volts to it and the IOH on the classy...alleviates it a little, but not near the amount that you will be penalized.

[[XC] gomeler]

dude, the moment I saw the SR-2 and storage-oriented in the title I immediately thought of the slot latency due to the NF200. You need to use a Rampage III Extreme for this build as it has a single X58 chipset and splits the 4 slots into x8 bandwidth which is the maximum each card can physically handle. Major bummer but at least you didn't buy a second Xeon. Try to return the SR-2 and hopefully get your money back(or sell it on a forum for ~80% retail).

It is a shame you live in NZ otherwise I'd loan you my ES RIIIE to make sure your cards function properly. Try to get in touch with Deanzo, he lives in NZ and maybe if you are lucky and he is willing he can work something out to test such a setup for you to save you the trouble. Otherwise find a normal X58 board with 3 slots and use a PCI video card for testing purposes.

Your test setup looks very similar to my test setup at work by the way Be very careful with those Intel drives. I hammered one of them with IOMeter random 4kb 10/90 write/read and another with random 4kb 100/0 write/read for a few weeks and both are effectively dead. My 10k rpm SAS HDDs are now faster in random write performance The OCZ Vertex LE I tested held up much better but it also degraded. At least SSDs are progressing fast enough that by the time I kill them the next revision is out.


edit: I'm not 100%, Lardarse would be the best person to ask, but you should see some performance gains out of overclocking the PCIe bus. Try 110MHz or 115MHz and see if the controllers or VGA hiccups. I don't know if the NF200s use a separate PLL or use the PCIe frequency, I suspect it operates based on the PCIe frequency, but it doesn't hurt to try.

[AceNZ]

You need to use a Rampage III Extreme for this build as it has a single X58 chipset and splits the 4 slots into x8 bandwidth which is the maximum each card can physically handle.

I need more RAM than the X58 boards can handle--and I also want ECC, which they don't support.

The alternative is probably a server board. The ones I've found with enough x8 slots don't also have an x16 for the video. However, I've seen some benchmarks that show current-generation video cards with x8 on PCIe 2.0 performing within a few percent of x16, so it might work out OK.

Try to return the SR-2 and hopefully get your money back(or sell it on a forum for ~80% retail).

It's past 30 days now, so probably too late to return the SR-2, but I'll look into it. If not, resale should still be pretty easy.

Be very careful with those Intel drives. I hammered one of them with IOMeter random 4kb 10/90 write/read and another with random 4kb 100/0 write/read for a few weeks and both are effectively dead.

That's quite a pounding; if I was intentionally going to try to wear out a drive, I would do something close to what you did. I normally only run read benchmarks on my drives. Of course, iometer does a write for the initial setup, and there are writes for creating a new filesystem, etc, but the load is relatively light, particularly when spread over 16 drives.

Also, I'm only using 64GiB on each SSD, which helps minimize write amplification problems.

edit: I'm not 100%, Lardarse would be the best person to ask, but you should see some performance gains out of overclocking the PCIe bus. Try 110MHz or 115MHz and see if the controllers or VGA hiccups. I don't know if the NF200s use a separate PLL or use the PCIe frequency, I suspect it operates based on the PCIe frequency, but it doesn't hurt to try.

Yep, that's the plan. I'm also wondering whether a faster QPI would help.

[[XC] Oj101]

Sure. Is there anything in particular that you'd like to see (or hear)?

You'll give us two minutes of video? Well, installation of the following:

Windows 7 64 Bit Ultimate
Office 2010 Professional
Photoshop CS5
...
[skip list of 400 other huge applications]
...
Just Cause 2
Metro 2033
Crysis

And the remaining 40s can be for booting

[Sandon]

I have the no overclocking server version of this board (a bit different but very similar otherwise):

http://www.newegg.com/Product/Produc...-175-_-Product

I really like it. Since I have a lot of storage and use ECC memory I didn't want to risk doing any OC in the first place.

Very nice build but I think I would have leaned more towards this SM case as it is very nice:

http://www.newegg.com/Product/Produc...282&Tpk=7046GT

(that is the bare bones kit)

Even the case alone is not cheap though.

[[XC] Oj101]

I just choked when I saw your Speedtest result, WWWWWWWWWTTTTTTTTTTFFFFFFFFFFF????!!!!

[AceNZ]

I updated the Intel firmware -- no noticeable change in performance.

I moved one of the 9260s from an x8 slot to an x16/x8 slot, next to the other 9260 (which is in an x8 slot, since the x16/x8 next to the video card is blocked by the audio cable).

The 4KB random performance unfortunately went down as a result. Sigh.

I'm mostly using two iometer tests: 4KB random reads with 2W, QD128 each; 128KB sequential reads, 1W, QD128; both with a 1GB test file. I generally used a 32K strip size. I tried a number of different configurations; here are my best numbers so far:

Single drive:
random: 38378 IOPS
sequential: 260 MBps

So far, so good: both random and seq are above Intel's specs of 35000 and 250, resp.

The "x" multipliers below are factors compared to a single drive.

Bootable:
random:
-- 8 drives: R80 --> 145916 IOPS (3.8x)
-- 16 drives: 8R0 * 2 S/W R1 --> 174186 IOPS (4.5x)
sequential:
-- 8 drives: 8R0 --> 1570 MBps (6.0x)
-- 16 drives: 8R0 * 2 S/W R1 --> 2305 MBps (8.9x)

Not bootable:
random:
-- 16 drives: 8R0 on each of 2 controllers --> 194639 IOPS (5.1x)
sequential:
-- 16 drives: 8R0 on each of 2 controllers --> 3144 MBps (12.1x)

While it's disappointing that 8 drives on one controller don't scale to 8.0x, the shocking thing is that going from 8R0 on one controller to 8R0 on two controllers (8 vs. 16 drives) only results in a 33% performance improvement, when it should be an easy double since there's no additional RAID logic or per-controller load.

My hope for this project was at least 200K IOPS (5.2x) for 8 drives, with near-linear scalability for each added controller. The NF200s seem to be the most likely culprit. Damn.

[Sandon]

I just choked when I saw your Speedtest result, WWWWWWWWWTTTTTTTTTTFFFFFFFFFFF????!!!!

Hehe, I work for a web-hosting company that has 30 gigabits of connectivity which is fun to do testing from =)

[AceNZ]

I have the no overclocking server version of this board (a bit different but very similar otherwise):

http://www.newegg.com/Product/Produc...-175-_-Product

I really like it. Since I have a lot of storage and use ECC memory I didn't want to risk doing any OC in the first place.

That's the exact SM board I was looking at. I almost got it while I was waiting for the SR-2 to become available, but held off because it doesn't have an x16 slot; they're all x8. Silly me.

Very nice build but I think I would have leaned more towards this SM case as it is very nice:

http://www.newegg.com/Product/Produc...282&Tpk=7046GT

(that is the bare bones kit)

Even the case alone is not cheap though.

I have a SM case for my current server. It's very nice. However, it's also very, very LOUD. I removed several of the fans and cranked the speed on the remaining ones way down, and it's still loud. The current machine will be used as a desktop box, so the usual SM cases won't work, although they do have a "quiet" line that might be worth looking into.

[Sandon]

That's the exact SM board I was looking at. I almost got it while I was waiting for the SR-2 to become available, but held off because it doesn't have an x16 slot; they're all x8. Silly me.

Yeah that was the one downside for me too but when I looked up testing I found that PCI-E 2.0 x8 compared to x16 caused almost no slowdown (only a couple percent max) even with the most recent video cards. This wansn't the case with PCI-E 1.0 so I went ahead and got the board and have been very happy with it. There is also this board I was originally looking at but it wouldn't fit my case:

http://www.newegg.com/Product/Produc...82E16813182176

The problem is that extra 1.5 inches or whatever near the top. It would fit that SM case but not likely the case you have.

I have a SM case for my current server. It's very nice. However, it's also very, very LOUD. I removed several of the fans and cranked the speed on the remaining ones way down, and it's still loud. The current machine will be used as a desktop box, so the usual SM cases won't work, although they do have a "quiet" line that might be worth looking into.

Well I will agree with this but with the newer SM boards and newer revisions of the case its a lot better as you don't have to run the fans at full speed. I know on my 2U server that its actually not *that* bad in volume when I have the fans set down to a low speed (and temps are still acceptable) but I can understand that they likely are not an option if you need something that is really really quiet and loudness is of course all relative. What might be not that loud to me could be way to loud for you =)

Of course now I don't really care (noise) as I am just gonna have all my machines in a rack that is in another room and long cables going through the closet into my room for my monitor and stuff:

[Tiltevros]

I updated the Intel firmware -- no noticeable change in performance.

I moved one of the 9260s from an x8 slot to an x16/x8 slot, next to the other 9260 (which is in an x8 slot, since the x16/x8 next to the video card is blocked by the audio cable).

The 4KB random performance unfortunately went down as a result. Sigh.

I'm mostly using two iometer tests: 4KB random reads with 2W, QD128 each; 128KB sequential reads, 1W, QD128; both with a 1GB test file. I generally used a 32K strip size. I tried a number of different configurations; here are my best numbers so far:

Single drive:
random: 38378 IOPS
sequential: 260 MBps

So far, so good: both random and seq are above Intel's specs of 35000 and 250, resp.

The "x" multipliers below are factors compared to a single drive.

Bootable:
random:
-- 8 drives: R80 --> 145916 IOPS (3.8x)
-- 16 drives: 8R0 * 2 S/W R1 --> 174186 IOPS (4.5x)
sequential:
-- 8 drives: 8R0 --> 1570 MBps (6.0x)
-- 16 drives: 8R0 * 2 S/W R1 --> 2305 MBps (8.9x)

Not bootable:
random:
-- 16 drives: 8R0 on each of 2 controllers --> 194639 IOPS (5.1x)
sequential:
-- 16 drives: 8R0 on each of 2 controllers --> 3144 MBps (12.1x)

While it's disappointing that 8 drives on one controller don't scale to 8.0x, the shocking thing is that going from 8R0 on one controller to 8R0 on two controllers (8 vs. 16 drives) only results in a 33% performance improvement, when it should be an easy double since there's no additional RAID logic or per-controller load.

My hope for this project was at least 200K IOPS (5.2x) for 8 drives, with near-linear scalability for each added controller. The NF200s seem to be the most likely culprit. Damn.

Try to make 8 Raid's of 0 config with 64k stripe size.
R0 with 2 drives each and then when u will make 8 pairs raid them with windows. Then try to test it and see ur performance.

In IOmeter try to test 1MB file with 64QD. with only one worker for ur maximum speed.
Now u have a 12 Thread Cpu. To test well the 4k file u should test it on 3-4 workers. what i mean??? set to worker 1 to 3 or 4 the target and then run it. U will see the sun shine
Good luck with ur setup.
Im using a dual 5520 chipset and belive me with one 9260i8 with fastpath and 8 intels im hitting 200.000+ IOPs.
Tip of the day disable the windows diffender when u r running benchmarks.

[AceNZ]

Try to make 8 Raid's of 0 config with 64k stripe size.
R0 with 2 drives each and then when u will make 8 pairs raid them with windows. Then try to test it and see ur performance.

Results:
4K random: 175741 IOPS (about the same as 8R0 * 2 SW R0)
128K seq: 2437 MB/s (about the same as 16 * SW R0)

With 8 drives on one controller configured this way, random is 139083 IOPS, which is about the same as 8R0.

In IOmeter try to test 1MB file with 64QD. with only one worker for ur maximum speed.

Using the same 2R0 * 8 S/W R0 array suggested above:
Result for 4K random, 1W: 141230 IOPS

Now u have a 12 Thread Cpu. To test well the 4k file u should test it on 3-4 workers. what i mean??? set to worker 1 to 3 or 4 the target and then run it. U will see the sun shine

For the same 1MB file above and the same volume as above:
4K random, 3W: 207208 IOPS
4k random, 4W: 201225 IOPS

With 8 drives on 1 controller (2R0 * 4 SW R0), I get about 140K IOPS.

Im using a dual 5520 chipset and belive me with one 9260i8 with fastpath and 8 intels im hitting 200.000+ IOPs.

So with a 1MB test file, I can get the same 200K IOPS, but it takes two controllers with FastPath and twice the number of drives.

[Anvil]

Ace,

Did you try 8-16KB stripe size?

[AceNZ]

Did you try 8-16KB stripe size?

Yes. Here are the results by strip size:

rnd, 4K, qd128, 2W, 16K, 8R00, 111590 IOPS
rnd, 4K, qd128, 2W, 8K, 8R0, 113833 IOPS
rnd, 4K, qd128, 2W, 64K, 8R0, 129823 IOPS, always read-ahead
rnd, 4K, qd128, 2W, 16K, 8R0, 131150 IOPS
rnd, 4K, qd128, 2W, 32K, 8R0, 138815 IOPS
rnd, 4K, qd128, 2W, 64K, 8R0, 144314 IOPS

seq, 128K, qd128, 1W, 8K, 8R0, 1127 MB/s
seq, 128K, qd128, 1W, 16K, 8R0, 1234 MB/s
seq, 128K, qd128, 1W, 64K, 8R0, 1263 MB/s
seq, 128K, qd128, 1W, 16K, 8R00, 1369 MB/s
seq, 128K, qd128, 1W, 32K, 8R0, 1570 MB/s
seq, 128K, qd128, 1W, 64K, 8R0, 1612 MB/s, always read-ahead

R00 is building single-drive "spans" in MSM, then combining them with R0.

In general, I found that S/W R0 is faster than H/W R0, and H/W R1 is faster than S/W R1 (although I'm interested in the fastest config, eventually I'll need some kind of redundancy).

[Anvil]

Ace,

Not bad at all, I was hoping for a bit more though.
A bit strange that the 64K strip is the better performer, that might be due to the FP Key.
Did you clean the drives between each bench or are the drives in "steady state"?

I still haven't received my FP key (delayed for another week), I might start playing with my 2 LSI 9260s without the key just to get to know how they perform together.
Each 9260 is ~90' iops (w/o the key) so in theory they should do 180' iops, looking at your numbers that might not be the case.

[AceNZ]

Not bad at all, I was hoping for a bit more though.

Me too!

A bit strange that the 64K strip is the better performer, that might be due to the FP Key.

64K is only better for random, or for sequential when read ahead is enabled (RAE). The hit for RAE is so large on the random side, though, that to me, the best combined random and seq throughput was 32K -- that's what I used for much of my other tests.

Did you clean the drives between each bench or are the drives in "steady state"?

Steady state. Cleaning (secure erase) requires the drives to be connected to the ICH10R, so it's not easy with my current setup. Also, most of my tests are for reads only, so a dirty drive shouldn't matter.

I still haven't received my FP key (delayed for another week), I might start playing with my 2 LSI 9260s without the key just to get to know how they perform together.
Each 9260 is ~90' iops (w/o the key) so in theory they should do 180' iops, looking at your numbers that might not be the case.

For obvious reasons, I would be very interested in your two-controller results, even without FP. Would you do those tests on your UD7?

[AceNZ]

I tried one more slot-swap, moving one card into the x16/x8 right next to the paired x8 slot (so the last three slots have the LSI cards in them). The result was a small (3%) improvement in 4K random to 200525 IOPS for 8R0 * 2 drives.

I'm coming to the end of my RMA / return period for the SR-2, so I decided to file the paperwork for a return. However, while waiting for EVGA to approve it, I thought of one more thing I should try: using Windows Server 2008 R2 Enterprise instead of Win 7 Ultimate x64.

Using Win 2008, seq perf was the about the same as Win 7. Random perf for 8 drives was about the same as Win 7, but was 247303 IOPS for 16 drives -- that's 23% better than Win 7! It's only 1.8x the single-8R0 performance, so short of 2.0x, but not nearly as bad as before.

I'm still trying to decide if I'm going to keep the SR-2; in addition to the difference between 1.8x and 2.0x, there's the single-8R0 perf, which is only about 3.6x a single drive.

[Anvil]

Ace

Yes, I'll be doing the tests on the UD7.

I might be able to start testing this weekend.

edit:
Interesting results using Windows 2008 Server R2.

23% increase is quite a lot, much better.
Wonder what's wrong with W7, which drivers did you use?

[AceNZ]

Interesting results using Windows 2008 Server, R2?

Yes, R2 Enterprise.

23% increase is quite a lot, much better.
Wonder what's wrong with W7, which drivers did you use?

I ran a preliminary test with the driver that came with Windows, and it was terrible -- about half the final result, which came with the latest 9260 driver from LSI. I actually haven't installed the chipset drivers yet....

I'm not sure why Win 7 should be any slower. I know the Enterprise edition has some NUMA optimizations that aren't available in other versions of Windows, but with only a single CPU, it's hard to imagine how that would help. I also know Microsoft cleaned up the interrupt handling code in Windows Server as part of a performance improvement effort directed around supporting 10 Gbps networking. Maybe it helps other devices too?

I used Windows Server 2003 as a desktop OS for more than a year while I put off making the switch to Vista--the server platforms are workable, but there always seem to be minor gotchas around things like missing or incompatible drivers, and apps that will only run under a client OS or where they charge 10x the price for the server version.

[Anvil]

I've got a few computers running Windows Server 2008, none of them using SSDs.
I switched to W7 when the Beta was released and so I've never really used R2 as an OS except for in VMs and on one of my laptops.

The server editions are quite nice once you get them tuned for workstation usage.

I noticed Tilt mentioned that one should disable the Defender while doing tests, did you try that as well?

[AceNZ]

I noticed Tilt mentioned that one should disable the Defender while doing tests, did you try that as well?

On W7, yes, Defender is disabled; I also don't have anti-virus software installed. Defender doesn't seem to be installed by default with W2K8.

It's probably worth another pass at lightening W7 before giving up on it.