Build log: storage-oriented SR-2

[tiro_uspsss]

I ordered a new MB today. The main factors were:

-- 47% scalability for 4K random with 8R0 on one controller (146K IOPS measured with FP vs. 200K target vs. 307K @ 100% scalability)
-- 40% scalability for 4K random with 8R0 on two controllers (16 drives)
-- 41% improvement in sequential throughput for Solaris72 with the change to a non-NF slot
-- 17% less for 4K random with 6 drives and no FP, compared with Anvil
-- Results from Tilt and others with non-NF boards that are showing much better scalability (a demo at Intel with a direct 5520-based MB showed close to 100% scalability for 28 SSDs)

Latency is clearly the performance killer: on the controller itself we can see it when the FP key is added: 78% improvement for 8R0 on one controller--and at the MB level from the NF200.

I'm disappointed to lose to the ability to OC (an easy 25% gain in CPU perf), but I'm not willing to lose 40 to 60% of my I/O performance in exchange; I'd rather buy a faster CPU or add a second CPU instead.

My feelings about the SR-2:

-- The support for OC-able dual Xeons is really at the heart of the board
-- Seems like it would be great for quad-SLI, with 2 to 4 SSDs on the ICH10R
-- Great as a OC'd cruncher
-- Nice to have support for eSATA, SATA 6.0, USB 3; although their presence isn't a make-or-break factor for me
-- Great to have 6 RAM DIMMs per CPU, including ECC support
-- Some issues with MB layout are a pain: the audio cable's location blocking one of the I/O slots, non-standard mounting holes (the plastic standoffs don't cut it), a non-standard power LED connector (2 pins instead of +/NC/-).
-- It would also be nice if the next version of the MB was about a cm or so narrower, so that it doesn't block the I/O cage for large add-ons like the 5-drive bay I'm using; I'd be willing to give up any or all of the extra add-on devices in exchange
-- And of course the key thing for me is that I would love to see the dual-NF200s replaced by dual 5520s

some of the slots on the SR-2 have lanes 'purely' from the 5520, right?
if so, are they affected by the NF200 latency?

which mobo u going to get?
can I recommend this one? :

http://www.newegg.com/Product/Produc...82E16813182174

[tiro_uspsss]

oh & what are u going to do with ur SR-2?

[Nizzen]

some of the slots on the SR-2 have lanes 'purely' from the 5520, right?
if so, are they affected by the NF200 latency?

which mobo u going to get?
can I recommend this one? :

http://www.newegg.com/Product/Produc...82E16813182174

Or this with LSI SAS 2 integrated

http://www.supermicro.com/products/m...0/X8DTH-6F.cfm

[tiro_uspsss]

Or this with LSI SAS 2 integrated

http://www.supermicro.com/products/m...0/X8DTH-6F.cfm

... thats the same mobo..

[jmaas]

Greetings,

I have been following this discussion with great interest, I am very interested to know if you can tell me which motherboard you replaced the SR2 with? I would like to complement you on your tesning and research you have done.

I would like to send you a document from Intel about SSD Perfromance (it's some what dated and you may have already read) I will send you a PM.

Thanks for your efforts. Very interesting read.

Cheers

Justin

[AceNZ]

some of the slots on the SR-2 have lanes 'purely' from the 5520, right?
if so, are they affected by the NF200 latency?

No, the SR-2 has dual NF200s. Since each NF200 has two x16 inputs (with one x16 output to the 5520) and since the x8 slots are shared with the neighboring x16, all slots go through the NF200s.

which mobo u going to get?
can I recommend this one? :

http://www.newegg.com/Product/Produc...82E16813182174

That's exactly the one I ordered yesterday; I've had my eye on it for months now.

oh & what are u going to do with ur SR-2?

I have an RMA number, so I'm planning to ship it back to EVGA for a refund.

Or this with LSI SAS 2 integrated

http://www.supermicro.com/products/m...0/X8DTH-6F.cfm

Yep. I got that model -- otherwise, the eight PCIe lanes that are associated with the onboard controller aren't accessible.

I have been following this discussion with great interest, I am very interested to know if you can tell me which motherboard you replaced the SR2 with?

As above: SuperMicro X8DTH-6F, which has dual 5520s, seven true x8 slots, up to six DIMMs per CPU, integrated LSI SAS2008 6Gbps x8 controller, dual Gb LAN, onboard VGA, ICH10R with 6 SATA ports, and support for IPMI / KVM-over-IP. No onboard audio.

I also looked at the same Tyan MB as Tilt, but I prefer seven x8 slots and six DIMMs per CPU to the S7025's four x16 slots and four DIMMs per CPU--although the Tyan's probably a better choice for those who are more focused on video performance than I am.

[tiro_uspsss]

No, the SR-2 has dual NF200s. Since each NF200 has two x16 inputs (with one x16 output to the 5520) and since the x8 slots are shared with the neighboring x16, all slots go through the NF200s.



That's exactly the one I ordered yesterday; I've had my eye on it for months now.

that sucks

nice pick!

[AceNZ]

Here's a block diagram of the X8DTH-6F:



It's shocking to me that EVGA can't or won't publish this type of block diagram for the SR-2.

Also -- what's a good, free video benchmark? I'd like to compare the x16 with NF200 to the direct x8 for my video card.

[Movieman]

I think your going to find that the dual 5520 chipset SM board will present it's own issues with latency but I wish you luck with this.
I have this sitting in my living room but damned if I can get it to post with any chips I've tried inc the earliest X5570's( Bios text says it was tested with those chips at the beginning) to X5670's..
Board lights up,fans spin but nothing else..
http://www.supermicro.com/products/s...7046GT-TRF.cfm

[AceNZ]

I think your going to find that the dual 5520 chipset SM board will present it's own issues with latency but I wish you luck with this.

Perhaps, although Tilt and Intel have shown encouraging SSD perf results with dual 5520 boards.

I have this sitting in my living room but damned if I can get it to post with any chips I've tried inc the earliest X5570's( Bios text says it was tested with those chips at the beginning) to X5670's..
Board lights up,fans spin but nothing else..
http://www.supermicro.com/products/s...7046GT-TRF.cfm

Sounds annoying. Some SM boards have a reputation for being picky about RAM -- I imagine you've tried different sticks already....

Does the board have IPMI? If so, have you tried to force a power-on reset that way? I know of at least one SM board where that worked when the normal POST sequence failed.

[Movieman]

Perhaps, although Tilt and Intel have shown encouraging SSD perf results with dual 5520 boards.



Sounds annoying. Some SM boards have a reputation for being picky about RAM -- I imagine you've tried different sticks already....

Does the board have IPMI? If so, have you tried to force a power-on reset that way? I know of at least one SM board where that worked when the normal POST sequence failed.

Like Isaid,I wish you luck.I've seen how much you have into this in not only $$ but in time.
Memory is Kingston DDR3-1333 ECC and works fine in my SM X8DA3 so should be good on this one.
IMPI,maybe..Didn't check as I don't normally use it.
What I need to do is a bios update but can't even get a post.
Board has onboard vid.Tried that and 2 vid cards and jumpers were in correct positions..
Very strange,usually SM boards are tanks but this one frustrated me.

[Tiltevros]

Like Isaid,I wish you luck.I've seen how much you have into this in not only $$ but in time.
Memory is Kingston DDR3-1333 ECC and works fine in my SM X8DA3 so should be good on this one.
IMPI,maybe..Didn't check as I don't normally use it.
What I need to do is a bios update but can't even get a post.
Board has onboard vid.Tried that and 2 vid cards and jumpers were in correct positions..
Very strange,usually SM boards are tanks but this one frustrated me.

0,06-0,05ms are good for u???
No latency in storage as far i can say for the 5520 from Tyan.
Now if u want to hit low latencies in 5520 Tylesburg u should be very carefull in bios settings. Low Power and ECO things are adding more latency but when u disable them u will hit Raw performance.
The S7025 is not for Video performance is just for Teslation and that is the reason that i am using it.
Wow i forgot to tell u.. Just put a fun on 5520's they r getting to warm. It would be cool if u find any watercool heads for Tyles.
Pm me when u got it to help u out. Wellcome to dualist gang

[AceNZ]

Very strange,usually SM boards are tanks but this one frustrated me.

Have you checked the CMOS battery to make sure that it's still good (usually 3 VDC)? An old battery can prevent a normal power-up and POST.

Do you have a speaker on the board (sometimes requires a jumper change to enable it)? If so, are you getting any beep codes?

[Movieman]

Have you checked the CMOS battery to make sure that it's still good (usually 3 VDC)? An old battery can prevent a normal power-up and POST.

Do you have a speaker on the board (sometimes requires a jumper change to enable it)? If so, are you getting any beep codes?

Never thought on the battery, BUT I think it's a video issue.
Sort of a moot point now as I don't have any old Gainestowns to boot it's old bios with, just have newer 32nm Westmeres here.

[AceNZ]

The video should include sound and a general "tour" around and inside this "monster"

Here's a link to a short video of the system:

http://www.youtube.com/watch?v=_KxM2W0063E

A 480p version is available too. Sorry for the poor lighting in the beginning.

[Tiltevros]

beatyfull work mate hope u enjoy it.

[AceNZ]

Here's Cinebench results:

[Anvil]

Thanks for the video Ace, much appreciated!

Looks like I'll have to order more drive cages for my SSDs.
(I've only got one drive cage, got it this weekend and it certainly makes it a lot cleaner, the rest of my drives are just laying around in the case )

Looking at your video one shouldn't believe that you've got 16 SSDs there, all due to the drive cages, keeps the cable count to a minimum)
I'm using a model from Chieftec and they've just released a new model that fits 6x 2.5" drives in a single 5 1/4" bay that I might order, I just need to make sure that it supports 6Gb/s drives.
Link to Chieftec

[AceNZ]

The Chieftec looks nice, although it looks like it takes 6 data cables -- one thing I like about the SuperMicro bays is that they only need one cable for power and one for data to support 4 drives.

[Computurd]

how are you connectin gthose drive bays to the raid cards, one data cable? is that sas to sas?

[AceNZ]

how are you connectin gthose drive bays to the raid cards, one data cable? is that sas to sas?

Yes, one data cable for each 2.5-inch drive bay (4 drives). The connectors are SFF-8087 on the raid card side, and SFF-8484 on the drive bay side:

http://www.newegg.com/Product/Produc...82E16816103189

The 3.5-inch bay uses five mini-SAS style connectors.

[Anvil]

The Chieftec looks nice, although it looks like it takes 6 data cables -- one thing I like about the SuperMicro bays is that they only need one cable for power and one for data to support 4 drives.

Yeah, I'm aware of that , it saves quite a bit on the power cabling though.

[AceNZ]

The new motherboard arrived yesterday. Re-build is in progress.

[Computurd]

excellent news! cant wait to see the results

PS...do not flash to the new firmware for 9260 it is borking controllers left and right!

[AceNZ]

Here's the X8DTH-6F with the CPU, heatsink and RAM installed:



Close-up of the heatsink:



Note that I'm using 6 RAM sticks now: 6 * 4 = 24GB.

The board's mounting holes lined up well with the ones on the PC-P80, with the exception of the one in the upper right, which happened to be the same place that the SR-2 needed one of those adhesive plastic pins. So, I left that one in place, and re-used it for the X8DTH, and removed one other. In the process, I found that the adhesive on those pins is amazingly strong; my earlier worries about them possibly coming loose were clearly unfounded.

The board is now up-and-running. One detail for people not familiar with server boards: the BIOS comes pre-configured to use the VGA port on the back panel as the console. On first boot, nothing is displayed on the video card. I had to hook up a VGA display and change the BIOS settings from "Onboard VGA" to "Offboard VGA" to have it use the main video card as the console.

I'm not using the onboard LSI controller yet, and in fact had to disable it in order to have enough option ROM to boot from the HDDs. It uses two SFF-8087 connectors that you can see in the lower-right corner of the board.

The HDD configuration was preserved with the switch-over. In fact, I didn't have to reinstall Windows; the OS was smart enough to reconfigure itself (as a former OS developer, I found that to be very impressive).

The video card is too big to fit in the usual top slot, closest to the socket for the second CPU (it hits the RAM). The bottom three slots are on the 5520 that's directly connected to the first CPU, so I put the video card in the third slot to hopefully minimize latency. The RAID cards are in slots 1, 5 and 7 (numbered from the bottom up; pics to follow).

It's not an SR-2 any more, so maybe I should start a new thread...