Fastpath..

[dwayneberry]

Hi,

The configuration I am running is 3 lsi-9260-8i each running raid 5 on 8 X25-M G2 drives. (a total of 24 drives). The three lsi raids are then combined using on OS (windows 2008 R2) stripe. It is 3.36TB in size.

I need the array to have redundancy (in case of disk failure) but also very fast, and large enough to fit a serious amount of data. This config also allows for potentially 3 disk failures (assuming different controllers).

I have several other raid setup using Areca 1231ML and 1680-xi but only the LSI controllers are currently capable of getting close to pushing the capabilities of the drives connected. On the Areca 1680-xi with 16 fast SAS drives the card is completely overwhelmed. That was why I went to the multiple cards when moving to trying to utilize the SSD drives.

The machine is a dual Xeon 5520 with 48GB mem.

The performance is great but was really hoping for more on the random access, so am hoping to see good fastpath numbers (based on what I have seen on this and other forums I am hoping for something around 4X... am I crazy? )

P.S. If I increase the read size I can get to over 4gb/s

[AceNZ]

@dwayne -- besides R5, how do you have your controllers and volumes configured? Stripe size?

[Computurd]

you will see tremendous improvements, with a array of that scale and complexity i would highly recommend you purchase the keys. the results with raid 5 are supposed to be just as impressive. i was wondering, is it possible to run a raid 0 set across all three controllers as a bootable array? i know that arrays can be spanned over several controllers, just not sure if they are bootable or not.
i would love to see some pics of your rig!

[dwayneberry]

Here are some pictures.

I know it looks bad!! It is in a better box now 2U supermico case with 24 2.5 drives across front, but it took months for the box to become available so it ran on a frankenhack setup for most of its life.





I do not have a link to a photo of it in the box but here is a link to the manufacturer site

http://www.supermicro.com/products/c...16A-R900LP.cfm

It was a squeeze to get it all in the box and if I had to do it again I would use the 9261 cards due to where the cables connect, as it is I had to very carefully butcher each cable to get its bend just right to fit cables in tight box.

Unfortunately the box is an active box in real R&D use so I cannot just re-purpose the drives. I played with raid 0 a little initially (just out of curiosity) and saw better numbers than the raid 5 but for the benefit of the redundancy etc the gain was not at all worth the risk. To be brutally honest with any kind of disk setup like this it is very difficult to push the "potenital" versus the real world use. When I am say doing oracle testing on this machine I might see peaks of usage in the 1gb/s kind of range but often it sits down around 600mb/s if I deliberately up the amount of work being asked to be done on the box thought I get very nice scaling.

I am really looking at the random io benefits of the fastpath key to truely fully realize the potential of the box.

[trans am]

godamn it dudes and friends in this thread! you know me as trans am my friends know me as alvin.
it doenst matter. I have gone through hell and back with the 9260-4i on every platform . amd sucks and is hit or miss depending on board. So after I got the lsi it made me move back to intel. but i always stuck to what I know. 2 ssds in raid0 off lsi and a F3 off intel for storage.
Ive gone through 2 ocz samsung ssds then 2 indinlinx barefoot supertalent ssds to currently 2 50gb vertex 2 ssd. then last weekend one of the vertex2 died. so i rma to ocz.

lemme get this straight cuz I feel like 2 ssd with the fp key is not taxing the controller the limits. Ive invested so much time and money into this controller but I still feel like my current setup is not maxing it out to its full potential/


what can I do with this setup? should I get 2 more 50gb ocz vertex 2 and stripe them so I use all the 4 sas/sata connections. or should I shut the hell up and just wait for the rma v2 to come back and just do what I have been doing....2x ssd raid0 and run the f3 off the board?

I feel like Im not using the card to its limits and as a ln2 overclocker I want to push hardware to hell and back. I feel so stupid with this post but Ive just had it. Ive invested way too much time and gone through 3 generations of ssds since i got the damn thing. I am running out of space on the F3 1tb drive as my storage. should I get a 2nd f3 and raid them as a 2nd raid or should I get 2 more 50 gb ssd vertex 2 and stripe them all together then run a 2nd 1tb f3 as a redundant set off the board or stripe them? anything you guys want to add Im all ears. Ive fing had it at this point. I should be benching this 655k on ln2 and break some hwbot records but im sitting here trying to figure out the best setup with the 9260-4i instead. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[GullLars]

Where's packman?

To answer your question TA/alvin, 4R0 C300 256GB (or 128GB) would take the controller close to the IOPS limit, and also give good bandwidth and storage space, but cost a lot. If you flip some pages back you will find graphs i've made, and you'll see the C300 does well at 3-4R0 with little IOPS scaling with more than 4 drives.
4R0 C300 128GB would give you:
512GB storage, about 150K IOPS read and 120K IOPS write, 1200-1400MB/s read, 500-550MB/s write. The real bonus with 9260 FP + C300 is really good IOPS scaling at lower QDs.

[mbreslin]

My advice: Wait for rma to return, use your two drives in r0, not a weak setup by any means. Then, wait patiently for intel refresh, possibly get a few gen3's.

[AceNZ]

i was wondering, is it possible to run a raid 0 set across all three controllers as a bootable array? i know that arrays can be spanned over several controllers, just not sure if they are bootable or not.

Dynamic disks configured as a software mirror / R1 are bootable. Dynamic stripes / R0 are not.

One problem is that you can't install W7 onto a dynamic striped array, nor can you convert a regular drive to a striped drive; you have to create it as a striped drive from the start.

[SteveRo]

@Ace - Paul might be referring to the idea that multiple controllers in the same machine could communication with each other prior to the mobo looking for an os to boot.
Would be cool if it could be done.
Might need to be an sli like connector between the controllers?

[trans am]

Where's packman?

To answer your question TA/alvin, 4R0 C300 256GB (or 128GB) would take the controller close to the IOPS limit, and also give good bandwidth and storage space, but cost a lot. If you flip some pages back you will find graphs i've made, and you'll see the C300 does well at 3-4R0 with little IOPS scaling with more than 4 drives.
4R0 C300 128GB would give you:
512GB storage, about 150K IOPS read and 120K IOPS write, 1200-1400MB/s read, 500-550MB/s write. The real bonus with 9260 FP + C300 is really good IOPS scaling at lower QDs.

thanks MB and Glrs. I think I will wait for the rma to come back as I am not about to blow cash on those c300s. If anything I was considering adding a 3rd 50gb vertex2 and raiding 3 but can I stripe across 3 disks?


or how about this.?? I noticed my 1tb f3 is almost maxed out for storage. could I get a 2nd f3 with the same model id./ no. and run the 2x50 vertex set raid 0 and the 2 f3 off the other 2 sata/sas ports in raid0? or is that just retarded to run your backup disks in raid0? crap..
that would also mean I would need a 3rd 1tb drive to back up to in order to copy my existing files to the new raid set.

any advice?

[Computurd]

thanks steve, that is exactly what i am referring to. you can hook some controllers together for a "teaming" effect via sas-to-sas cabling. 9260 comes with a sas-to-sas cable so it seems to be common sense. did some more research you can do this with a 9260 and one other controller, but not another 9260 (BOOOO!!)
you can do it with a 8080 (why only a 3gb/s i will never understand)
it is possible they just dont allow it because it would cut into their expander buisness is my best guess. there are controllers from other companies that you can span arrays across. just not sure if they are boortable or not....and im not worried about any 3 gb.s stuff anyways.
getting ready for 12gb/s myself
(2012 btw is release date for dual core 300,000 iops 12gb/s device from lsi...it is pcie 3.0 !!)(they are already sampling to select OEM's btw)

[GullLars]

I've known about the dual core thingy for a couple of months now, and seen a nibble of info on it. If i remember correctly it was up to 300K IOPS on integrated RAID, and up to 600K IOPS in pass-through softRAID.

[Tiltevros]

Tip of the day.

Try to marge some 9280 with some pass through devices witch they will have some external and internal ports
set the raid on the 9280 and then u will have some ROC's on passthrough and one Main ROC for raiding

[lowfat]

If you are still looking for iometer results here are mine.

http://www.2shared.com/file/PWN7QiHk...e_nocache.html

[Computurd]

Holy e! that QD of 1-4 is uber dude! hey Gullars could you add that to our graphs for comparison? nice results lowfat, thanks!

[lowfat]

Holy e! that QD of 1-4 is uber dude! hey Gullars could you add that to our graphs for comparison? nice results lowfat, thanks!

I didn't have Excel installed so I haven't even seen the results.

[Computurd]

this is random 4k right?

here is some of your numbers
QD MB/s
1 59.75
2 99.85
3 113.06
4 114.00
etc... that is about the highest it goes, but the point is it does it at such low QD..the highest is a 127@QD27 btw

[lowfat]

this is random 4k right?

here is some of your numbers
QD MB/s
1 59.75
2 99.85
3 113.06
4 114.00
etc... that is about the highest it goes, but the point is it does it at such low QD..the highest is a 127@QD27 btw

I used the config file posted by GullLars here

I didn't change any settings.

[GullLars]

Yes, those are 4KB random ioXtreme numbers.
I'm a bit surprised by the limited scaling, but you have awesome IOPS at low QD, actually better than SteveRo's Acards.

QD 1: 15,3K IOPS
QD 2: 25,5K IOPS
QD 3: 28,9K IOPS
QD 4: 29,1K IOPS
QD 27: 32,5K IOPS

Clearly the ioXtreme hits a wall around 30K IOPS, which is a bit sad. If it hadn't it could likely have done around 100K IOPS at QD 7-8. Still, for desktop usage the stats are awesome taking the bandwidth into consideration too.
15K IOPS @ QD 1 with MLC flash

[Computurd]

oh yea those are uber, could you put up a low QD graph (1-4 or 1-8) with his device against our recent fastpath arrays? man if only it would BOOT i would buy it right now!

[SteveRo]

content moved to the correct thread - G's rantings

[Solaris72]

I got my fastpath key today for my lsi 9260-4i card and installed. I didnt see any difference at all and hdtune shows the same result with and without fastkey? Is there anyway i can confirm that the key is properly installed and working??
Please help.

[trans am]

I got my fastpath key today for my lsi 9260-4i card and installed. I didnt see any difference at all and hdtune shows the same result with and without fastkey? Is there anyway i can confirm that the key is properly installed and working??
Please help.

http://www.xtremesystems.org/forums/...&postcount=243

http://www.xtremesystems.org/forums/...&postcount=241

did you install the drivers? did you install from a previous image? did you try a different bench like atto or as ssd or cdm?

[Solaris72]

Yes. I installed latest drivers and firmware.

Please find enclosed fw log from the megacli. Is my ssd is not supported by fastpath?

I'm seeing this lines here...is this okie?? i'm worrying about the bold ones...

T3: Primary Key: Security disabled
T3: Upgrade Key: Security is not supported even though Upgrade Key was detected
T3: Primary Key: EnableSnapshot disabled
T3: Upgrade Key: Snapshot is not supported even though Upgrade Key was detected
T3: Primary Key: CTIO disabled
T3: Upgrade Key: CTIO is supported
T3: Primary Key: SSC disabled
T3: Upgrade Key: SSC is not supported even though Upgrade Key was detected

[AceNZ]

How many SSDs do you have on the controller?

In my testing, I didn't notice much of difference in IOPS until I got to about 4 drives / QD32 / 80K IOPS. Supposedly latency is lower at lower QD, but I didn't track that.