Build log: storage-oriented SR-2

[AceNZ]

The SSDs are all 80GB X25-M G2s. 10 are Intel brand and 6 are Adata; 16 total (1280GB). The Adata's are also made by Intel, just re-branded with a new sticker and box (they can also be a little cheaper, depending on promotions, etc). Here's the packaging:



After inserting them into the 2.5-inch drive trays. They're attached with four small flat-head screws. The screws sit in the same slot that's used to hold/align the tray when it goes in the bay, so it's important to make sure they are tight and flat.

I don't care for the big blue Adata labels, but there's a note on them that says "warranty void if removed."



The drive trays slide into the bay and lock into place.



I pretty much finished up the power and data cabling today. More pics later, but one issue I ran into is that the Enermax molex connectors have some problems. Everything else about the power supply has been great so far, but I had a very tough time plugging-in the molex connectors. I had to re-align and slightly widen the pins with a jeweler's screwdriver in a couple of cases. Also, they have plastic flaps on the top and bottom, presumably to make it easier to remove them. However, those flaps also push against the drive bay cases, making it so they don't stay seated.



My solution was to bend the flaps back before inserting.

[Anvil]

WOW

This should be an interesting build.

When do you plan on doing the initial testing?

[lowfat]

16 SSDs!?!?!1

[InCredible]

jesus..i got one intel x25-m and im happy as hell..this is ...well xtreme haha

[mbreslin]

Must resist buying 24 port 1880 and maxing it. :x

[AceNZ]

When do you plan on doing the initial testing?

Assembly is almost done. If all goes well, later today I hope to do powerup testing and install Windows 7.

[theystolemyname]

When you run crystaldiskmark on those 16 intel x25-m g2s in raid 0, I imagine it will just say "you win" instead of displaying scores.

[Anvil]

AceNZ,

Really looking forward to your first impressions and a few benchmarks

What did you find out about the FP Key?

That build deserves a minute or two of video footage

[AceNZ]

What did you find out about the FP Key?

In what respect?

I would have liked to use it on the 9261 (for cabling reasons), but haven't yet been able to figure out where to hook it up.

That build deserves a minute or two of video footage

Sure. Is there anything in particular that you'd like to see (or hear)?

[Anvil]

Ace,

In respect to the 9261 controller, I guess you could wait for the software version of the FP Key.

The video should include sound and a general "tour" around and inside this "monster"
Maybe some footage while running benchmarks, it's up to you. I'll gladly watch whatever you'd like to show.

[AceNZ]

I'm using six 1TB WD RE3 drives. Here are the first five in their trays:



After installing the trays into the 3.5-inch drive bay:



Here's the back of the drive bay, showing the SATA and power cables:



For the sixth drive, I'm using the mini-rack that came with the PC-P80. The default configuration allows drives to be loaded only from inside the case. The wiring in the case is too dense once everything is cabled for that to be workable, so I reversed the rack so it loads from the front.



Unfortunately, when the rack is reversed, it blocks the lower black grill:



The cables on the 9260 controllers are SFF-8087 on the controller side, and SFF-8484 on the drive bay side -- one cable handles four SSDs.



With the back-to-back LSI controllers and their (potentially explosive) Li-ion batteries, I'm a little concerned about the heat, so I decided to use the 12 cm fan that came with the Enermax power supply. I took one of the long bolts that came with the PC-P80 that's intended to help position the plastic support for heavy video cards (I'm using one already for the HD5850), and used it to hold the fan to the chassis. With a little cable re-routing from the 9261 (and its damn 1 meter long cables), it looks great. In that location, it should also help to pull some air through the HDDs when the front door of the case is open (such as when burning DVDs). I could get power (and an RPM reading) from the SR-2, but decided to hook it to the HDD drive bay instead. Otherwise, I would have to change a jumper on the drive bay so that it doesn't sound an alarm when the fan is missing.



I'm still not happy with the way the plastic adhesive standoff on the motherboard is working. I added a little strain relief on the front and back of the SR-2. I don't like it, but I guess it's better than keeping the case laying permanently sideways. If anyone has any better ideas, please let me know!

The strain relief here is the white plastic tie that's looped through the heatsink:



The strain relief here is the black twist-tie that's connected to a hole in the corner of the SR-2:

[WhiteFireDragon]

wow you have an INSANE storage set up. what will the computer be used for?

[AceNZ]

I ran into a few glitches with the power-up check: forgot to remove a few jumpers to silence the no-fan alarm on the 2.5-inch drive bays, and one DOA WD RE3 drive. Also, for some reason the third controller isn't visible in the BIOS as a boot option--not sure if it's an LSI or EVGA problem.

I used the LSI WebBIOS to configure the controllers: 8R0 for each of the 9260s, with 8 SSDs each, and 2*3 R50 for the 1TB RE3 drives on the 9261.

After that, I installed Win 7 Ultimate -- no problems (took about 23 minutes from start to finish), although W7 created a strange 100MB partition on the second SSD volume, which messes up my benching plans a little. I'll probably re-install soon.

Here's E-LEET:



Regarding the earlier question about how well the heatsink is working, here are temps after being up for a couple of hours. Ambient is about 17C at the moment.



Initial bench results were terrible. I upgraded controller firmware and drivers, which helped tremendously. I also configured a software mirror / R1 between the two controllers. I'm not sure I'll keep it this way, but I was curious to confirm that it's bootable (it is), and to see the performance.

The numbers still aren't where I'd like them to be on the sequential side, but here's a quick CDM teaser:



I noticed that during the CDM sequential test, only half of the drives are active. The app is issuing single-threaded 1MB reads. The OS could in theory split up large reads between the two halves of the mirror, but it unfortunately doesn't. The small strip size probably also has an impact.

One artifact of the bootable dynamic disk / mirror is that it seems to re-sync when I reboot, which can take quite a while, not to mention the extra write wear on the SSDs.... Not sure yet how viable that will be for the long-term.

wow you have an INSANE storage set up. what will the computer be used for?

Thanks. It will be a desktop machine, not a server -- so things like benching, demos, software development and a little gaming.

[Anvil]

I guess you should keep on experimenting with the settings on the controllers, not sure about raid-1 and I'm pretty sure I'd end up splitting the drives into several arrays.

I almost always end up using NRA, DIO, WB or WT, DCE.
(wb or wt depends on what drives I'm using, wt is great for SF based drives, wb seems to works best for the Intel drives)

I would suggest you get a separate boot drive while testing/exploring the arrays, that way you can reconfigure the array(s), e.g. changing stripe size without reinstalling the OS.
(Get yourself 2 X25-Vs as bootdrives, they're terrific as boot drives)

1 drive DOA out of 22 ain't that bad, it could have been worse.

I'm using my rig for software development as well. (mostly VS and a bit of 32bit standard windows as well, running each environment in separate VMWare Workstation guests)
I change my config from time to time, that's why I'm not booting on the array, I like to reconfigue and do tests when I can.

[AceNZ]

I guess you should keep on experimenting with the settings on the controllers, not sure about raid-1 and I'm pretty sure I'd end up splitting the drives into several arrays.

I have many tests planned before deciding on a final config.

I'm starting to wonder if I should update the firmware on the SSDs.

I almost always end up using NRA, DIO, WB or WT, DCE.
(wb or wt depends on what drives I'm using, wt is great for SF based drives, wb seems to works best for the Intel drives)

By NRA, do you mean "no read ahead"? The available settings seem to be "Normal", "Read ahead" and "Adaptive". I have it set on "Normal."

Is DCE "drive cache enabled"? I have them set on "drive cache unchanged" at the moment.

The controllers all have BBUs, so I'm planning to use WB (unless there's a performance penalty, which is hard to imagine in this case).

I would suggest you get a separate boot drive while testing/exploring the arrays, that way you can reconfigure the array(s), e.g. changing stripe size without reinstalling the OS.
(Get yourself 2 X25-Vs as bootdrives, they're terrific as boot drives)

I was planning to use the HDD array as a boot drive during initial testing, but unfortunately it doesn't show up in the BIOS. I updated the controller firmware already, and that didn't help; I'm going to update the SR-2 BIOS today, to see if that does the trick.

Since I can't update the SSD firmware or run the SSD toolbox on drives while they're attached to the RAID controllers, I'm thinking of adding another 2.5-inch drive bay that would be connected to the ICH10R ports. Could be used as a short-term boot device during testing, too.

[Anvil]

I expect you have a test plan

What firmware is on the drives? 02HD is the latest from Intel.

NRA=No Read Ahead, DCE = Disk Cache Enabled (policy)

If you've still got Adaptive your not on the latest firmware/bios or MSM on the 9260, Adaptive is obsolete, the only ones left are No Read Ahead and Always Read Ahead.
I've been using WB all along on the Intels, I've performed a few tests and read iops are up a bit using WT, the penalty lies in writing of course, you just have to try for yourself.

On my UD7 I can select what raid card I wish to boot from, I've got one 9260 and one 9211, both are selectable. (dont know what would happen if I added a third card)
You could try to switch posistions for the 9261 and the first 9260, that should make the HDDs bootable. (if the HDDs are on the 9261 that is)

Getting another drive bay connected to the ICH is a great idea.

[AceNZ]

What firmware is on the drives? 02HD is the latest from Intel.

I can't tell yet, due to the "direct-connect" issue I mentioned earlier.

If you've still got Adaptive your not on the latest firmware/bios or MSM on the 9260, Adaptive is obsolete, the only ones left are No Read Ahead and Always Read Ahead.

Strange. I'm running the latest firmware and MSM, downloaded yesterday from the LSI site:

Firmware 4.4.1 / version 12.7.0-0007 (APP-2.70.03-0862), released 26 May 10
MegaRAID Storage Manager 4.4 / version 6.90-05, released 6 May 10

You could try to switch posistions for the 9261 and the first 9260, that should make the HDDs bootable. (if the HDDs are on the 9261 that is)

I thought of that. It's a hassle from the cabling side, but might be worth it if the SR-2 BIOS update doesn't fix it.

[Anvil]

If you power down and look at the drives the FW is printed on the label on the drives. (top right)

As for the latest LSI fw/bios/MSM, that is indeed strange, Adaptive was removed in April this year iirc. (I've got the same fw/versions installed)

Did you notice what versions the cards lists when you boot?

[AceNZ]

After updating the SR-2 BIOS:

1. I can still only see two volumes in the boot menu, but they were two different ones than with the older BIOS: one of them was the RAID-50 array.

2. I can no longer run the LSI WebBIOS array manager. The clunky CLI interface still works. The fact that the CLI interface exists at all tells me that I'm probably not the first person to run into this.

3. I noticed a message before the OS boots that says something along the lines of "not enough space to load option ROM."

4. I was unable to boot W7 from the existing array.

So, I reinstalled W7 on the R50 array, which is what I wanted originally anyway, until I understand the performance quirks of the SSDs a little better.

After switching to DCE mode, here are the results from iometer (4K random, QD=256, 2W) and CDM, using the mirrored dynamic disks. The numbers still seem much lower than they should be.

If you power down and look at the drives the FW is printed on the label on the drives. (top right)

Thanks. It's also displayed by the LSI BIOS as it boots. Most of the drives have the latest FW, but a few don't. Time to rig up a way to update the older ones...

As for the latest LSI fw/bios/MSM, that is indeed strange, Adaptive was removed in April this year iirc. (I've got the same fw/versions installed)

I couldn't re-run the WebBIOS to double-check, but I'm pretty sure that's where I saw Adaptive. It's not present in MSM.

[LardArse]

you are running out of shadow memory, u can disable some boot roms (lan etc etc and lower memory low gap to reserve more space

[AceNZ]

you are running out of shadow memory, u can disable some boot roms (lan etc etc and lower memory low gap to reserve more space

I already disabled USB 3, SATA III and the rear eSATA ports, and that didn't help (although it does improve boot time slightly). I'll double-check, but IIRC, LAN boot is disabled by default.

Is there anything else that I missed?

What's the "memory low gap"? Is that an explicit setting?

[Computurd]

"memory low gap" is basically an amount of memory allocated for pci-e slot usage. uber fast gpu's, sli and tri-sli setups need higher memory low gap settings. and of course raid cards. i always set mine at the highest allowed as i run a sound card, tri-sli, and a x8 raid card. i am not sure of the implications of usage with three raid cards as per your setup. and this is in laymans terms btw. not sure if im exact in my description.

[AceNZ]

I set the memory low gap to 1GB, which was the lowest value available. I still only see two of the three boards in the list of bootable devices, but the change did allow the LSI WebBIOS to run again. There is what looks like an error message that's displayed right before the OS boots, but it flashes so quickly I can't read it.

Here's a pic of the WebBIOS startup screen that shows the FW versions:



And here's one that shows the read-ahead options, and that "Adaptive" is still there:



I started running a series of SSD benchmarks yesterday. I'm making progress, but one surprising discovery was that random read throughput almost doubles when I switch from write-back to write-through caching.

[Johnny87au]

Try updating firmware on the intel SSD's, i know when i raided 2x the scores were average, updated with the firmware tool and bang my read/write etc was skyrocketing..

[AceNZ]

Try updating firmware on the intel SSD's, i know when i raided 2x the scores were average, updated with the firmware tool and bang my read/write etc was skyrocketing..

I definitely will update the FW; just have a bit of a cabling issue to handle first.

In the mean time, I found that the drives with the old FW are the six drives from Adata (02HA); the ten Intel-branded drives are all current (02HD). I had the Adatas on a separate controller from the Intels, so I could bench them separately; their performance was about the same.

I get 146K 4KB random IOPS from either 8R0 array with 1W/QD128. However, if I run 1W on both 8R0 arrays at the same time, througput only goes up to about 195K. If I run 128KB sequential requests on both arrays at the same time, total bandwidth comes close to doubling. My conclusion is that there may be a latency issue of some kind at the motherboard level.

I'm beginning to think the OC-ability of the SR-2 may not be worth the tradeoffs they made with the I/O architecture (dual NF200 and one 5520 vs. dual 5520) -- although I do understand why they made the choice they did (to make quad SLI possible; but I'm not using SLI). The 5520 only has 36 PCIe lanes. The SR-2 claims to support 6*16=64 lanes, but they are actually muxing through dual NF200s.

Each NF200 muxes two x16 connections into a single x16. Since EVGA says that using the x8 slots next to the neighboring x16 causes that slot to also run x8, it's clear those pairs are also linked. What's not clear is which slots go to which NF200.