Build log: storage-oriented SR-2

[AceNZ]

I'm working on a storage-oriented SR-2-based system. It will end up being a little unusual, so I thought the build log might be of interest.

Here's the SR-2:

sr2box-front.jpg

sr2box-back.jpg

sr2bag.jpg

sr2-board.jpg

This is the contents of the box. The version I received didn't come with the mini-debug board, just a photo and a note saying that they will send them out later.

sr2-contents.jpg

After opening the bags with the cables, etc (excluding the SLI parts, which I won't be using). The paper under the user manual opens up into a large quick-start guide. The small bag above the red and black cables contains the plastic adhesive standoffs that are needed for some cases.

sr2-open.jpg

[metalop1g]

I like storage builds, they tend to use kinda pricey hardware

[saint-francis]

How is this going to be storage oriented? The SR2 is oriented toward raw computational power. Wouldn't you be better off with a SM board with 1 socket and an Areca 1880 loaded up with 1 TB SCSI drives with the OS on a RAID 1+0 array of 4 X25M's?
Last I heard the SR2 didn't even work with ECC RAM.

EDIT:

Sorry. I don't mean to all over your obviously awesome new build. It's just not what I would do, even if I shat golden eggs and Tiffany cuff links.

Honestly, I don't know why all RAM isn't ECC.

[mbreslin]

@saint-francis: You're selling 1880s already cool I'll take one, thanks.

[AceNZ]

I'm using a single X5650 CPU for now, with 3 * 4GB Kingston DDR3 1333 ECC RAM and a Noctua NH-U9DX heatsink with dual fans:

cpu.jpg

Why the UD9X? Since the SR-2 will be mounted vertically, I wanted to minimize the weight of the heatsink, to keep stress on the board to a minimum. The UD9X only weighs about 500 grams, compared to 1000 grams or more for some of the larger heatsinks. I'm only planning a mild overclock, so the smaller size should be OK. To partly compensate, I'll add a second fan, in a push-pull configuration.

In addition, I wanted to minimize amount the heatsink overhangs the RAM sockets.

heat1.jpg

heat2.jpg

I'm using 4GB RAM sticks. The RAM is ECC, which is important to me, and is one reason for preferring Xeon-based motherboards over Core i7. Unfortunately, the fastest ECC sticks I could find were 1333.

ram.jpg

The heatsink will mount in either orientation (fans on the left and right or on the top and bottom). I chose left and right to correspond with the direction of airflow in the case. This also minimizes the overhang on the RAM sockets.

mounted.jpg

The UD9X didn't completely eliminate the RAM socket overhang, but it's not too bad, as you can see below; it's just the bottom fan clip, and there might be enough room to squeeze a stick into that first socket. Not shown here is the fan clip on the top side of the case, which came into contact with a heatsink on the SR-2 at first; I had to bend it up slightly to prevent that.

over.jpg

[AceNZ]

How is this going to be storage oriented? The SR2 is oriented toward raw computational power.

Raw computational power and storage are not incompatible.

Also, the SR-2 has much more going for it than just compute power. You'll see what I mean later in the thread.

Wouldn't you be better off with a SM board with 1 socket and an Areca 1880 loaded up with 1 TB SCSI drives with the OS on a RAID 1+0 array of 4 X25M's?

I looked at SM, but the OC-ability of the SR-2 is a plus for me.

The 1880's aren't available yet. Even when they are, who knows what the perf will be compared to existing options -- not to mention firmware maturity, frequency of updates, etc, etc.

Last I heard the SR2 didn't even work with ECC RAM.

Where did you hear that? Support for ECC RAM is listed in the SR-2 specs. Several ECC modules are even included on their list of officially-supported modules (though no 4GB sticks are listed yet).

[=[PULSAR]=]

Can't wait to see the end product Ace What kind of drives and controllers are you going to run?

[AceNZ]

I'm using the Lian-Li PC-P80 case.

li-case-1.jpg

li-case-2.jpg

li-case-3.jpg

li-case-4.jpg

There are three 14 cm fans inside the front door:

li-case-5.jpg

12 drive bays, along with a fan speed control at the top:

li-case-6.jpg

10 PCIe slots at the back, along with a 12 cm exhaust fan:

li-case-7.jpg

The case uses a removable motherboard tray. You can see a small box of parts taped to the inside with blue tape:

li-case-8.jpg

On the top of the case: the power and reset buttons, four USB ports, eSATA, Firewire (not available on the SR-2, unfortunately) and audio ports.

li-case-11.jpg

Parts inside the box (I didn't use the extra mounting plate):

li-case-12.jpg

[Movieman]

subbed..

[slaveondope]

Shiney
Very nice, jealousy's a b**** I would know

[Deux]

I can confirm that the SR-2 supports ECC ram, running 12 GB in mine not 4GB sticks though.

Also, I'll be interested to hear your thoughts on the heatsink, I've got a pair on my 5677s and so far am not very impressed. The mounting pressure seems to be very lacking in my opinion. Cool build you've got going here, looking forward to following along

[AceNZ]

Can't wait to see the end product Ace What kind of drives and controllers are you going to run?

Mainly X25-M drives and 9260-8i controllers -- with a few twists to keep things interesting.

[AceNZ]

When mounting the SR-2 on the motherboard tray, EVGA suggests using at least 9 mount points. Unfortunately, there are only 7 that line up with the tray in the PC-P80. EVGA includes a set of plastic mounts with adhesive backing. I used three of them along the bottom edge of the SR-2, and two on the upper right. Due to concerns about the adhesive interfering with the alignment of the holes, I only used the adhesive on 4 of the five mounts; I excluded the one on the lower-left corner. It's presence is still useful, though, as support against PCIe board mounting pressure. For the mount on the lower-right corner, I had to cut off part of one edge so that it would sit level on the tray (there's a wire-routing hole in that part of the tray).

One of the mount holes is located behind the plate that covers the south bridge heatsinks. The plate is held on with four tiny screws. Here's what it looks like behind the plate (the mount hole is to the lower right of the fan):

hidden.jpg

The approach I used to make sure the mount holes were lined up was to place a small block under the upper right side of the motherboard, to hold it slightly off of the mounting plate on that side. I lowered the board onto the plate and just started the screws in the mount holes on the left of the board, to get everything aligned. Then I removed the block and let the adhesive come into contact with the mount plate. PITA.

Here's the adapter plate for the SR-2 connectors. I removed the cover flaps over the blocked ports, since I'm planning to use them:

plate.jpg

After placing the mounted board in the case, I hooked up the power supply. I'm using an Enermax Galaxy Evo 1250W. At 240V (which we have here in New Zealand), it's 89% efficient at 50% load.

The one I got (from Newegg) came with a 12 cm blue LED case fan -- although I'm not planning to use it.

evo1.jpg

evo2.jpg

The video card is a Sapphire Toxic HD5850 with 2GB video RAM (plenty of "onboard storage"). It comes slightly overclocked from the factory.

video.jpg

Here's the inside of the case after hooking things up. I'm using the factory / stock cables, so wire management is always a challenge:

inside1.jpg

My first complaint about the SR-2: the audio connector is located right in-line with one of the PCIe slots. It looks like I won't be able to use that slot if I use the onboard HD audio:

audio.jpg

Here's the back of the system, showing the power supply, the motherboard adapter plate and the video card:

back.jpg

At this point, I hooked up a monitor and did a quick power-up check (there's a power switch on the SR-2, which made this a little easier). I got through POST and into the BIOS OK; the system found all 12GB of RAM, and everything looks good so far.

[WhiteFireDragon]

wow i'm surprised it fits in that PC-P80 case. so it doesn't officially fit, but you had to do some slight modding to get the board in there?

[mezcal]

Subbed.

[AceNZ]

wow i'm surprised it fits in that PC-P80 case. so it doesn't officially fit, but you had to do some slight modding to get the board in there?

The PC-P80 is on EVGA's "official" list of cases that will hold the SR-2. The mods were more in how the board is attached to the tray than to the case itself.

[AceNZ]

I'm using LSI storage controllers. Here's the packaging. What's really funny is the box on the right, which is how they ship a single (and very tiny) FastPath key.

lsi-boxes.jpg

Here's the 9260-8i, the FastPath key and the battery backup:

9260-parts.jpg

And again, after assembly (hat tip to @Computurd for the photo of how to hookup the FastPath; the included instructions reference a connector that doesn't exist). LSI sells the 9260-8i in two SKUs; one includes two one meter long SFF-8087 to mini-SAS cables, shown in the photo, and the other does not.

9260-assem.jpg

Repeat three times. One of the boards is a 9261-8i without FastPath. I ended up getting it mostly due to a combination of a stock shortage at Newegg when I placed my order and a slight doubt that 0.5 meter cables would be long enough in the PC-P80. It's functionally identical to the 9260-8i, but just has the SFF-8087 connectors on the back of the board instead of near the front (intended for use in low profile cases).

lsi-group.jpg

Install into the chassis. The slot next to the video card is indeed blocked by the audio cable. Assuming that everything works as expected, the new card should be fine in the next slot over, which is an x8. Using that slot prevents its neighbor from being used in x16 mode, but that's obviously not an issue here.

lsi-install.jpg

With three RAID controllers, I needed a motherboard that could support at least 16+8+8+8 = 40 PCIe lanes, which rules out vanilla X58-style boards. Most server boards also don't have the right number or type of lanes. Combined with my desire for ECC support, a 5520-based Xeon board like the SR-2 is really the best approach.

[Computurd]

holy mother of god!! jesus dude that is awesome! you should be able to span an array across all three raid cards with the 9260's and have a UBER raid 0 setup

[AceNZ]

Here are the 2.5-inch drive bays:

bay-front.jpg

There is a small and very noisy fan on the back. Fortunately, I don't need them here, and they're easy to remove. The connectors on the back are an SFF-8484 and one molex-style power connector; just two cables for all four drives.

bay-nofan.jpg

Repeat three more times (four bays total; 16 drives), and install in the chassis. I've also added a Pioneer BD-R in the top slot.

bay-install.jpg

Here is the 3.5-inch drive bay:

hbay.jpg

Since the case has a 14 cm fan that will blow air directly over the drives (when the door is closed), I've removed the fan on the back, which is sitting on top of the bay in the photo below. The bay takes two molex-style power connectors and five individual mini-SATA connectors.

hbay-back.jpg

It turns out that the bay is so long, and the SR-2 is so wide, that the bay was slightly covering the IDE connector on the SR-2. I'm not using any IDE drives, but I still didn't like the idea of partly covering the connector with a piece of metal. I found that if I removed the safety grill from the bottom fan in the door, that the door would close with the drive bay not quite hitting the connector; the extra mm or so was just enough.

Here's the bottom two fans, one with the grill and one without:

grill.jpg

Here's the bay after installation:

hbay-install.jpg

I was thinking about using a second 3.5-inch bay like this one, but it won't fit with the SR-2.

You can see how close the bay sits to the IDE connector in the photo below; it's just to the right of the SATA connectors. BTW, the bottom two red SATA connectors support SAS 6.0 using the Marvell chipset. I've connected the bottom one to the eSATA port on the top of the PC-P80.

hbay-ide.jpg

[[XC] Oj101]

I'm dreaming of 24 SSDs here

[Sam_oslo]

Looks like a great project.

I've been argumentation that SR-2 is a great option for Enterprise Server. It's good to see it shaping.

[SteveRo]

wow beautifull build so far!

[erikmorgan]

You got me! Sub'd!

[Chevelle]

Nice build. Can't wait to see more!

[Raychem]

Subbed.

ROFL...great animated avitar