Build log: storage-oriented SR-2

[Computurd]

you might also try disabling search functions as well...
however, of note here, is the server drivers for raid cards are almost always better imo. i use the 2008 R2 drivers with my 9260, all the time, everytime, even though i am running win7. you might try reinstalling win7 and using the server drivers anyway, several others with this controller also get better results this way. i wish you had a non-nf board handy so you could slap that 9260 on there and check it out. once you see the difference you would RMA that board STAT. night and day difference.
the high IOPS you are receiving may be nice, but what of the latency of those??? that is where you are getting killed. you might try running one of Gullars profiles that we ran to compare fastpath performance, then he will compare the IOPS/AccessTime against other single card results with these controllers, giving you a frame of reference on the acccesstime performance.
also your chipset drivers are bound to help, at least some, i would recommend you try putting those to use as well.

These comparisone here of iops/accesstime is quite ingenious and definitely denotes the QOS within the storage system, so i believe it would be there that the issue becomes more glaring. http://www.xtremesystems.org/forums/...&postcount=185

[AceNZ]

I've disabled search, too.

Interesting idea about the drivers; I compared the W7 and W2K8 versions, and the only differences are in the megasas2.cat files; the actual drivers, which are in megasas2.sys, are identical.

Yes, it would be wonderful to be able to do an A/B comparison with a non-NF200 motherboard. Unfortunately, I'm not so lucky. The question I'm struggling with at the moment is whether a potential 25%+ improvement in I/O performance is worth a definite 25% decrease in CPU and memory speed--plus, of course, the time, effort and cost to make the switch.

I'll run GullLars' iometer profiles to take a look at comparative latency; should be interesting.

[tiro_uspsss]

u've only got 1 CPU installed atm, right?

[AceNZ]

u've only got 1 CPU installed atm, right?

Right: one X5650 CPU and 3*4GB=12GB RAM.

[tiro_uspsss]

Right: one X5650 CPU and 3*4GB=12GB RAM.

did u have to place anything or do anything 'special' for the empty socket?

[AceNZ]

did u have to place anything or do anything 'special' for the empty socket?

No. The only things you have to do to use one CPU are to use the CPU0 socket and leave the plastic protective cover in place on the CPU1 socket.

I also went ahead and connected power for CPU1, but I don't think that's a requirement.

[tiro_uspsss]

No. The only things you have to do to use one CPU are to use the CPU0 socket and leave the plastic protective cover in place on the CPU1 socket.

I also went ahead and connected power for CPU1, but I don't think that's a requirement.

mm ok, thanks for the answers

[Anvil]

Ace,

My initial test using 4 X25-M G1, all drives are in steady state.

1x 2R0 on each 9260, stripe size 16KB, NRA, WT, DIO, DCD, no FP Key.
SW raid 0 W7 x64

iometer random read 4KB aligned, QD64, 1W
4r0_x25m_80gb_iometer.JPG

iometer random read 4KB aligned, QD128, 1W
4r0_x25m_80gb_iometer_qd128.JPG

iometer random read 4KB aligned, QD128, 2W
4r0_x25m_80gb_iometer_qd128_2w.JPG

[Computurd]

another thought occurred to me....could some of it be that you are only using one processor, therefore half of the available PCI-e lanes?

[lowfat]

another thought occurred to me....could some of it be that you are only using one processor, therefore half of the available PCI-e lanes?

Both CPU's share the same 5500 chipset. So this definitely isn't an issue.

[SteveRo]

@Mr Anvil - nice numbers!

[Anvil]

Steve,

Not bad for 4 drives and w/o the FP Key

I had to improvise a bit to use 6 drives. (most of my drives are in use)

4x X25-M 80GB G1
2x X25-M 160GB G2

2x 3R0 (2xG1 + 1xG2)

4KB random read, 4K aligned, 1W

[AceNZ]

another thought occurred to me....could some of it be that you are only using one processor, therefore half of the available PCI-e lanes?

Both CPU's share the same 5500 chipset. So this definitely isn't an issue.

All of the PCIe lanes are available to either CPU, but with two CPUs there are also two QPI links to the 5520 -- so I suppose it's possible that the increase QPI bandwidth might help improve throughput or latency (although there are potential memory/NUMA issues for the second QPI link).

Here's what a single 5520 architecture typically looks like without the NF200:

[Computurd]

gah it was QPI that i meant. came out as pcie...still confident the prob is nf200 though, the extra qpi isnt going to get rid of that latency hit

[Tiltevros]

this is how it looks like the S7025 as u can see 2x5520 chipsets with ESI connection with the ICHxR so the lanes are splitted.

Each CPU handles one 5520 chipset to split the lanes. thats why i have 72 lanes and a true PCIEx16 quadsli or quadfire

[xeon_1]

this is how it looks like the S7025 as u can see 2x5520 chipsets with ESI connection with the ICHxR so the lanes are splitted.

Each CPU handles one 5520 chipset to split the lanes. thats why i have 72 lanes and a true PCIEx16 quadsli or quadfire

Tyan still makes the best stuff in my eyes but so bloody expensive and hard to get a hold of.

[Computurd]

ace you should read this :
http://www.xtremesystems.org/forums/...d.php?t=254745

[AceNZ]

Thanks -- yes, very encouraging results in the other thread.

[Anvil]

Ace,

What's your conclusion?

I still haven't received the FP Key so I won't be able to show you any figures for a week or so.
The 173-175' iops I got using 6 drives is clearly limited by the standard controller without the FP Key, my 4 drive setup scaled close to 100%.

[AceNZ]

I've been sick the last few days, so my testing has been delayed. I hope to have some results / conclusions tomorrow.

My earlier tests on W7 showed the following scaling (with FP) with 1, 2 and 4 drives on one controller. In my notation, "S1" is short for "software RAID-1" and "1D" is short for one drive / volume.

rnd, 4K, qd128, 2W, --, 1D, 38378 IOPS (single drive = 1.0x)

rnd, 4K, qd128, 2W, 32K, 2R0, 71442 IOPS (1.86x)
rnd, 4K, qd128, 2W, --, 2S0, 72776 IOPS (1.89x)
rnd, 4K, qd128, 2W, 32K, 2R1, 73976 IOPS (1.92x)
rnd, 4K, qd128, 2W, --, 2S1, 74680 IOPS (1.94x)

rnd, 4K, qd128, 2W, 64K, 2R0*2S0, 75093 IOPS (1.95x)
rnd, 4K, qd128, 2W, 32K, 4R10, 93143 IOPS (2.42x)
rnd, 4K, qd128, 2W, 32K, 4R0, 97034 IOPS (2.52x)
rnd, 4K, qd128, 2W, 32K, 2R1*2S0, 104934 IOPS (2.73x)
rnd, 4K, qd128, 2W, --, 4S0, 106525 IOPS (2.77x)

seq, 128K, qd128, 1W, --, 1D, 260 MB/s (single drive = 1.0x)

seq, 128K, qd128, 1W, --, 2S1, 425 MB/s (1.63x)
seq, 128K, qd128, 1W, 32K, 2R0, 491 MB/s (1.88x)
seq, 128K, qd128, 1W, 32K, 2R1, 505 MB/s (1.94x)
seq, 128K, qd128, 1W, --, 2S0, 523 MB/s (2.01x)

seq, 128K, qd128, 1W, 32K, 2R1*2S0, 523 MB/s (2.01x)
seq, 128K, qd128, 1W, 32K, 4R10, 931 MB/s (3.58x)
seq, 128K, qd128, 1W, 32K, 4R0, 954 MB/s (3.66x)
seq, 128K, qd128, 1W, --, 4S0, 1043 MB/s (4.01x)

Summary: on one controller, sequential scaled well up to 4 drives; random only scaled up to two drives.

[Anvil]

rnd, 4K, qd128, 2W, --, 1D, 38378 IOPS (single drive = 1.0x)

rnd, 4K, qd128, 2W, 64K, 2R0*2S0, 75093 IOPS (1.95x)
rnd, 4K, qd128, 2W, 32K, 4R10, 93143 IOPS (2.42x)
rnd, 4K, qd128, 2W, 32K, 4R0, 97034 IOPS (2.52x)
rnd, 4K, qd128, 2W, 32K, 2R1*2S0, 104934 IOPS (2.73x)
rnd, 4K, qd128, 2W, --, 4S0, 106525 IOPS (2.77x)

Summary: on one controller, sequential scaled well up to 4 drives; random only scaled up to two drives.

So basically you're saying that you get less then 3x scaling using 4 drives? (random iops)

Do you have an X58 MB available?

[AceNZ]

So basically you're saying that you get less then 3x scaling using 4 drives? (random iops)

Yes; less than 3x with 4 drives on one controller.

Do you have an X58 MB available?

Unfortunately, no, not at the moment.

[Computurd]

would probably even be worth it testing on any recent mobo....like a 775 maybe?

[AceNZ]

Test results using Anvil's configs:

Using 4 X25-M 80GB G2, all drives are in steady state.

1x 2R0 on each 9260, stripe size 16KB, NRA, WT, DIO, DCD, no FP Key.
SW R0 W7 x64

iometer random read 4KB aligned, QD64, 1W


iometer random read 4KB aligned, QD128, 1W


iometer random read 4KB aligned, QD128, 2W


Then for 6 drives, as 3R0*2S0
iometer random read 4KB aligned, QD128, 1W


iometer random read 4KB aligned, QD128, 2W


Summary for 4 drives:
1W QD64: 3.3x single drive, 1.02x Anvil
1W QD128: 3.8x single drive, 0.99x Anvil
2W QD128: 3.9x single drive, 1.01x Anvil

For 6 drives:
1W QD128: 3.7x single drive, 0.83x Anvil
2W QD128: 4.6x single drive

I also did a quick comparison of FP vs. no FP:
8R0 on one controller, 32K strip, QD128, NRA, WT, DIO, DCE, 1W, iometer 4K random, sector aligned
no FP: 81931 IOPS
with FP: 145916 IOPS
--> 78% improvement

Sequential I/O is about the same with FP and without.

[AceNZ]

I ordered a new MB today. The main factors were:

-- 47% scalability for 4K random with 8R0 on one controller (146K IOPS measured with FP vs. 200K target vs. 307K @ 100% scalability)
-- 40% scalability for 4K random with 8R0 on two controllers (16 drives)
-- 41% improvement in sequential throughput for Solaris72 with the change to a non-NF slot
-- 17% less for 4K random with 6 drives and no FP, compared with Anvil
-- Results from Tilt and others with non-NF boards that are showing much better scalability (a demo at Intel with a direct 5520-based MB showed close to 100% scalability for 28 SSDs)

Latency is clearly the performance killer: on the controller itself we can see it when the FP key is added: 78% improvement for 8R0 on one controller--and at the MB level from the NF200.

I'm disappointed to lose to the ability to OC (an easy 25% gain in CPU perf), but I'm not willing to lose 40 to 60% of my I/O performance in exchange; I'd rather buy a faster CPU or add a second CPU instead.

My feelings about the SR-2:

-- The support for OC-able dual Xeons is really at the heart of the board
-- Seems like it would be great for quad-SLI, with 2 to 4 SSDs on the ICH10R
-- Great as a OC'd cruncher
-- Nice to have support for eSATA, SATA 6.0, USB 3; although their presence isn't a make-or-break factor for me
-- Great to have 6 RAM DIMMs per CPU, including ECC support
-- Some issues with MB layout are a pain: the audio cable's location blocking one of the I/O slots, non-standard mounting holes (the plastic standoffs don't cut it), a non-standard power LED connector (2 pins instead of +/NC/-).
-- It would also be nice if the next version of the MB was about a cm or so narrower, so that it doesn't block the I/O cage for large add-ons like the 5-drive bay I'm using; I'd be willing to give up any or all of the extra add-on devices in exchange
-- And of course the key thing for me is that I would love to see the dual-NF200s replaced by dual 5520s