X8DTH-6F, PC-P80, LSI RAID, many SSDs

[AceNZ]

(this is partly a repost from the end of the SR-2 thread)

I've decided to try the SuperMicro X8DTH-6F, with its dual 5520s, to see if it performs better than the EVGA SR-2 for this storage-oriented build.

Here's the X8DTH-6F with the CPU, heatsink and RAM installed:



Close-up of the heatsink and RAM:



Note that I'm using 6 ECC RAM sticks now * 4GB each = 24GB.

The board's mounting holes lined up well with the ones on the PC-P80, with the exception of the one in the upper right, which happened to be the same place that the SR-2 needed one of those adhesive plastic pins. So, I left that one in place, and re-used it for the X8DTH, and removed one other. In the process, I found that the adhesive on those pins is amazingly strong; my earlier worries about them possibly coming loose were clearly unfounded.

The board is now up-and-running. One detail for people not familiar with server boards: the BIOS comes pre-configured to use the VGA port on the back panel as the console. On first boot, nothing is displayed on the video card. I had to hook up a VGA display and change the BIOS settings from "Onboard VGA" to "Offboard VGA" to have it use the main video card as the console.

I'm not using the onboard LSI controller yet, and in fact had to disable it in order to have enough option ROM to boot from the HDDs. It uses two SFF-8087 connectors that you can see in the lower-right corner of the board.

The HDD configuration was preserved with the switch-over. In fact, I didn't have to reinstall Windows; the OS was smart enough to reconfigure itself (as a former OS developer, I found that to be very impressive).

The video card is too big to fit in the usual top slot, closest to the socket for the second CPU (it hits the RAM). The bottom three slots are on the 5520 that's directly connected to the first CPU, so I put the video card in the third slot to hopefully minimize latency. The RAID cards are in slots 1, 5 and 7 (numbered from the bottom up; pics to follow).

[Anvil]

Looking good

Onboard LSI 2008 8-Port 6Gbps SAS Controller (RAID 0, 1, 10; RAID 5 optional), a pity you cant use it. (it's probably an 9211)

How does the new MB compare in size to the SR-2?

[alfaunits]

I doubt the onboard LSI is 9211, since the board was released a year before the LSI 92xx series.

A pitty about the VGA, that sux...

[AceNZ]

Onboard LSI 2008 8-Port 6Gbps SAS Controller (RAID 0, 1, 10; RAID 5 optional), a pity you cant use it. (it's probably an 9211)

I can probably use it if I don't need to boot from my R5 array. It would be interesting to compare performance to the 9260.

How does the new MB compare in size to the SR-2?

It's smaller, both in width and height. There are actually a few inches now between the right side of the board and the drive bays, which is great.

A pitty about the VGA, that sux...

It's just a temporary cabling setup and a BIOS change -- no big deal.


Here's the board installed in the chassis (yes, I know the wire management could be better):



Here's a close-up of the cards. Notice the third LSI controller below the video card, and the black plastic support for the video card in the vertical metal bracket on the right side of the image.

The silver colored heatsinks on the motherboard are for the two 5520s.

[Johnny87au]

results results

[AceNZ]

I disabled C-State in the BIOS, and enabled high-power mode in Windows. For the W7 tests, I reinstalled Windows from scratch. For the W2K8 tests, I used the existing installation.

Cinebench:



About 1 fps faster than the SR-2 (1.7%), even though this board is using an x8 PCIe slot instead of x16. CPU results are about the same.

RAID config: DCE, WT, DIO, NRA
Test file size: 2,000,000 blocks (1GB)
Test duration: 3 sec warmup, 25 sec run
FastPath enabled on both controllers

iometer:

rnd, 4K, qd128, 1W, 32K, 8R0, sector aligned
SR-2, Win 7 Ult x64: 145916 IOPS
X8DTH, Win 7 Ult x64: 147645 IOPS
X8DTH, Win 2008 R2 Ent x64: 146581 IOPS

rnd, 4K, qd128, 1+1W, 32K, C2/8R0*2D
SR-2, Win 7 Ult x64, sector aligned: 194639 IOPS
X8DTH, Win 7 Ult x64, sector aligned: 196371 IOPS
SR-2, Win 2008 R2 Ent x64, sector aligned: 247303 IOPS
X8DTH, Win 2008 R2 Ent x64, sector aligned: 247592 IOPS
X8DTH, Win 7 Ult x64, 4K aligned: 198143 IOPS

seq, 128K, qd128, 1W, 32K, 8R0
SR-2, Win 7 Ult x64, sector aligned: 1570 MB/s
X8DTH, Win 2008 R2 Ent x64, sector aligned: 1576 MB/s

seq, 128K, qd128, 1+1W, 32K, 8R0*2D
SR-2, Win 7 Ult x64, sector aligned: 3144 MB/s
SR-2, Win 2008 R2 Ent x64, sector aligned: 3144 MB/s
X8DTH, Win 2008 R2 Ent x64, sector aligned: 2922 MB/s

Conclusions:

With 8 drives on one controller, for random performance the X8DTH is 1.2% faster than the SR-2 under W7. For sequential, they're about the same.

With 16 drives on two controllers, for random performance the X8DTH is 0.9% faster than the SR-2 under W7, and about the same under Win 2008 R2. On both boards, Win 2008 is 26% faster than W7. For sequential, the SR-2 is about 7% faster than the X8DTH.

Bottom line: the boards seem to perform about the same, with a slight edge on random for the X8DTH, and a more significant edge to the SR-2 for sequential. Using Win 2008 has a much larger impact than switching boards.

Well, I have to say that I hoped for much more. I'm going to try tweaking the BIOS settings to see if I can improve things a little.

Here are the CPU-Z reports:

[Anvil]

WCE = Disk Cache Enabled?

Could you try a run using WT, NRA, DIO, DCD (Disk Cache Disabled)

I wonder why it doesn't scale, I expect you didn't try the new FW?

[AceNZ]

Adding another worker per VD seems to solve the multiple controller scaling problem: 297016 IOPS (vs. 146581 for 1W and 8 drives).

Yes, right, WCE = disk cache enabled.

I tried DCD, and perf was pretty much the same (297007 IOPS for 2W on each VD).

I also tried enabling write back: 187948 IOPS, and cached I/O instead of DIO: 286460 IOPS.

Given the headaches some people are having with the latest firmware, I haven't tried it. I'm already running on the edge of available option ROM space.

I was running a stress test yesterday, and the overheat alarm went off. Realtemp shows that the CPU is in range, and the test wasn't using any PCIe devices, so I suspect RAM might be overheating, given the 6 sticks in close proximity. Time for another fan.

[Anvil]

Well, that's pretty close to 300.000 IOPS

The only issue I've got with the new firmware is that it won't let me enter the WebBios. (afaics)
I haven't spotted any changes in iops, so far.
Lets hope they'll fix it, quite a few using these controllers on non-server MBs.

[alfaunits]

Hmm, I couldn't find in the post, what controllers are you using?

OT: Do I understand right the IOPS results are with a 1 or 2 workers only at QD128? Asking because I couldn't get such high IOPS with a signle worker, but 4 workers made it.

[AceNZ]

Well, that's pretty close to 300.000 IOPS

Yes, but it takes two controllers. I thought the 9260 with FP was rated at 300K IOPS per controller.

Hmm, I couldn't find in the post, what controllers are you using?

Two LSI 9260-8i controllers with FastPath for the SSDs, a single 9261-8i for the HDDs.

OT: Do I understand right the IOPS results are with a 1 or 2 workers only at QD128? Asking because I couldn't get such high IOPS with a signle worker, but 4 workers made it.

I ran tests with 1W and 2W on one controller, and 2W, 4W and 6W on two controllers (1W, 2W and 3W for each controller). All of the tests were done with QD128.

The best perf came from 2W on each controller (4W total).

[AceNZ]

I rigged a fan to blow on the RAM:



Using prime95 for stress testing, everything works OK with the CPU/cache-only tests. However, the large FFT and balanced tests, which use RAM, still fail after just a few minutes: the on-board speaker sounds an alarm.

I'm not sure what's going on, but at this point I'm glad I haven't sent the SR-2 back yet....

[xeon_1]

i wouldn't send the sr-2 back atleast not until you get this one stable and if it overclocks.
If not i would go back to the sr-2 it might have higher latency but from what i gather its more stable and knowing evga it can take a beating.

[Computurd]

yes evga boards can take massive beatings. however, i am curious as to the latency differences, did you ever run one of gullars profiles, so maybe we could compare? what is the latency with your array?

[AceNZ]

i wouldn't send the sr-2 back atleast not until you get this one stable and if it overclocks.

The SuperMicro boards don't OC. That was my initial attraction to the SR-2: finally, an OC-able Xeon-based board.

however, i am curious as to the latency differences, did you ever run one of gullars profiles, so maybe we could compare? what is the latency with your array?

I ran GullLars' cached profile on the SR-2, but without FP, so I'm not sure how useful it is. Haven't done it yet on the new board.

I ran the Everest latency benchmark on the X8DTH, and it reported 0.06 ms -- the same as Tiltevros.

[Anvil]

This might be an interesting controller (PMC-Sierra PCIe 6Gb RAID Adapter)

Link to storagenewsletter

"PMC-Sierra's maxRAID architecture and next-generation RAID-on-Chip controllers will take future RAID 5 random read performance from today's 136K IOPS with the maxRAID BR5225-80 up to up to 300K IOPS on a single controller. A server configuration with four next-generation PMC-Sierra maxRAID adapters each connected to eight SSDs will achieve greater than one million IOPS."

[tiro_uspsss]

This might be an interesting controller (PMC-Sierra PCIe 6Gb RAID Adapter)

Link to storagenewsletter

"PMC-Sierra's maxRAID architecture and next-generation RAID-on-Chip controllers will take future RAID 5 random read performance from today's 136K IOPS with the maxRAID BR5225-80 up to up to 300K IOPS on a single controller. A server configuration with four next-generation PMC-Sierra maxRAID adapters each connected to eight SSDs will achieve greater than one million IOPS."

mmm.. didnt PMC-Sierra take over Adaptec?

[Anvil]

Yes they did.

Link to Adaptec

maxRAID just might fit into the naming of Adaptec's MaxIQ "technology".

[Anvil]

Yes, but it takes two controllers. I thought the 9260 with FP was rated at 300K IOPS per controller.

8x Intels (37.500 iops each) should in theory give you 300.000 iops per controller.

I don't know, one should ask LSI.

edit:
BTW, the Chieftec 4x2.5" "backplane" I ordered works for the 6Gb/s Crucial C300s.
(specifications indicate SATA 3Gb/s compatibility)

edit2:
iometer 4KB random read, 2R0 C300 tops my controller, ~89.000 iops, guess I'll have to wait for the FP key to see how far they go.

[AceNZ]

8x Intels (37.500 iops each) should in theory give you 300.000 iops per controller.

Yes. "In theory" being the key.

I don't know, one should ask LSI.

I remembered it wrong: the spec is up to 150K IOPS per controller with FP:

http://www.lsi.com/DistributionSyste..._PB_042910.pdf

That means the perf I'm seeing on both motherboards is actually in line with the specs.

[Anvil]

I've seen the 300.000 figure somewhere so that was my expectation as well.

This means that 4 Intels SSDs per controller is what you need or possibly 3x Crucial C300s.

[AceNZ]

This means that 4 Intels SSDs per controller is what you need or possibly 3x Crucial C300s.

Unfortunately, even with FP, the controller doesn't scale linearly beyond the first two drives, so with 4R0 I only see 97K IOPS, up to 119K with 4 * S/W R0.

[mbreslin]

Same server board, same drives, yet tilt submitted iometer results with 250k iops versus the 150-ish ceiling everyone else gets (and what's in the 9260 lsi spec sheet).

NDA, Secret Sauce, PCIE packet size.

Sure.

[Anvil]

Unfortunately, even with FP, the controller doesn't scale linearly beyond the first two drives, so with 4R0 I only see 97K IOPS, up to 119K with 4 * S/W R0.

Hmmm, as I said, I get 89-90.000 iops using 2x C300. (without the FP Key)

2R0 X25= ~75.000 iops, so with 2 extra drives, all you gain is 22.000 iops, a bit of a disappointment.

Maybe they'll release a FP key for the FP key, releasing all resources on the controller
There's certainly a lot of potential left in there somewhere, maybe in some other controller.

Well, ~300.000 iops isn't that bad especially considering the cost.

[AceNZ]

2R0 X25= ~75.000 iops, so with 2 extra drives, all you gain is 22.000 iops, a bit of a disappointment.

Gaining those 22000 IOPS requires 2 extra drives *and* the FP key.

You have to quadruple the number of drives and add the FP key to double the IOPS.

I'm starting to think that to maximize total throughput, I should move a couple of drives to the ICH10R.

Well, ~300.000 iops isn't that bad especially considering the cost.

True.