LSI 9260 settings

[IanB]

I know there are quite a few of you getting great results with the LSI 9260 controller, so I've bitten the bullet and ordered one, with 4 Corsair Force series 60GB drives. This is for a small network storage set-up, so it's a 4i controller and the drives will be in a RAID 10 configuration. The actual online storage needed is pretty low (all small files, mostly Word docs, little or no AV material) so I'm going to leave 10GB on each drive free for over-provisioning, ie. a 100GB array partition. I'm thinking about a 16K stripe. The machine has a UPS, but there won't be a battery-backup unit on the controller.

My question to those who have been using the 9260 for a while is: what are the best settings likely to be for this array on the controller? I mean read-ahead/write-back etc. Since this is a production machine, I don't want to spend time and reduce the drive lifetimes doing multiple benchmark runs for tweaking. This will be my first pro RAID controller, so I'd definitely appeciate someone talking me through the controller settings and any other setup tips I should be aware of.

Also some confirmation on the firmware issue would be handy - is the current version on the LSI website [12.9.0-0037 (APP-2.90.03-0933)] the last "good" version or the newest one that everyone was having trouble with?

[Computurd]

the current one listed is the version people are having problems with. That will depend upon your motherboard though. Which mobo are you running?
I would say that with your setup, the bandwidth of it being on a network access is going to be your greatest limiting factor, so the best settings would probably be direct I/O, with read ahead enabled. you are going to be running this in a server configuration, correct? So the only means of accessing the files to use them would be over ethernet? That will make for quite a solid setup!

[Anvil]

Is this for storage only or do you plan on installing the OS on the SSDs as well?

The 32KB stripe on the SF drives is the best compromise from my findings.

The rest of the settings are possible to change later.
I'd start using
- No Read Ahead
- Direct IO
- Write Back (as long as it's not mission critical and you do backups regularly)

If you're dealing with MS Office documents only, the SSDs are overkill imho, at the price I'd order the SSDs anyways
(You do get less heat, less noise)

[AceNZ]

I would recommend the same settings as @Anvil, except I wouldn't enable write back unless you have a BBU on the controller.

I would also set Drive Cache Disabled.

FWIW, for a fixed number of drives, I found RAID-5 to perform much better than RAID-10.

[IanB]

Yes, I should have been more specific about the hardware, sorry.

The network is around 35 XP clients running off Windows 2003 Server. Unfortunately the server is an old Socket 775 Dell Poweredge. It's got a Perc 5, but there's no way I want to add SSDs to that as I've not heard anything good about compatibility. Rebuilding the network server or moving large amounts of hardware around on it isn't an option right now as it's in continuous use, but the ultimate plan is to replace it with better hardware and move up to Server 2008 R2.

The new array is needed to provide all the user files, which were on a 10k SAS RAID 5 on the server Perc but two drives failed, so everything is currently running off a single SAS. Not good. Fortunately I had good backups, and the server OS is now on an SSD boot, so I just need to retire the final SAS and move those files somewhere better. At £200 per SAS drive, replacing with SSDs is economic and will provide way better performance and power economy. Plus it will improve my sanity, as my workspace is right next to this server.

My problem is the rest of the network hardware is very old tech so incremental upgrades are difficult and I'm fighting an uphill battle for funds to replace machines that still appear to "work" adequately. Much of it is DDR or pre-DDR hardware. There is only support for SATA2 on about half the machines. I don't have any machines on the network except one that has an ICH10R, and that is providing a RAID 1 backup share right now and doesn't have enough spare ports to add a further 4-drive array.

Since that is also the only machine with a x16/x8 PCIe2 interface (the server doesn't have a spare slot with the Perc in, and the server BIOS won't let me boot from any drive that isn't on the Perc ), that's the machine I'm adding the 9260 to. It's a Core i3 530 on a Gigabyte H57 uATX board, with a nice easy little overclock to over 3GHz. Sad to say, it's the best machine on the network. So this is (perhaps temporarily) becoming the main fileserver, but only as an XP client, not with a server OS. Obviously it has a Gigabit Ethernet port so that will be the limiting factor on network throughput. Having the array on the 9260 means I can easily move it somewhere more appropriate when the server hardware is upgraded, though.

So... no OS on this array. Just files. And user profiles. And of course they will be backed up very regularly. The Write Back setting is really going to be the one I have to consider carefully, safety versus speed...

FWIW, for a fixed number of drives, I found RAID-5 to perform much better than RAID-10.

OK, that's interesting. I was fairly sure that parity RAID would be a worse performer than RAID 10 for the same number of drives. My feeling was that I also gain a potential extra drive's worth of redundancy that way too.

The 32KB stripe on the SF drives is the best compromise from my findings.

Thanks. Again, my feeling was that since these are mostly small files, the best benefit would be a smaller stripe so that there is more chance of some parallelism in reads/writes. But 32K sounds good too.

There does seem to be some disagreement on whether Read Ahead is best set or not. Would that be a better setting if the files were larger, for longer sequential reads?

Thanks for confirming the firmware version, Computurd. Maybe I'll give it a miss. Is the previous working version archived anywhere, in case the version shipped is older?

Thanks for all your replies.

[Computurd]

the previous version will be on the same download page at lsi..up top there is a place to select 'current' or 'archived', archived being previous firmwares of course. with read ahead tbh i dont think it will make that much difference either way, but in that sort of environment i do not have enough experience to say either way for large files, yes it would be better. however, it can effect latency of other access and you will probably be receiving tons of simultaneous access with that many clients, when i originally read your post i thought this was more of a home user situation. like a WHS, definitely not 35 clients. you should get very good results with the 9260 though

[Anvil]

I agree with Ace, raid-5 should be an option to look into, you wouldn't notice any difference in speed (working with user files only) as the network would be the bottleneck.
If you were to run an sql server or some other sort of client/server setup, tweaking the VD config would be of more interest.

For raid-5 you could even use 1 of the drives as a spare, for automatic rebuild.

As for enabling/disabling the drive cache, I'm not sure, the SF drives do behave differently vs the Intels, either would work great though.

Utilizing 50GB per drive sounds like a good idea, the extra OP will make a difference in the end.

The BBU has been a pita for me, I've RMA'd mine, waiting for the replacement, as long as you've got the UPS you should be pretty safe, it's not an replacement for the BBU though.

[AceNZ]

Regarding stripe (really strip) size: it's impossible to say for sure in advance what will work best in your environment. However, in my testing on 4K random reads with 4 drives, I found 32K strip to be faster than smaller sizes. YMMV.

I've tried read-ahead in a variety of scenarios, and haven't yet found one where it helps.

There's a big difference between a parity RAID such as RAID-5 and a mirrored approach such as RAID-10: with parity, you can do data scrubs to actively correct errors; with mirrored drives, you can't. Mirrors only come into play from an error correction perspective if the other drive in the array reports an error. If bad data is returned with no error reported, then a mirror won't catch it (neither will a regular read of a parity array, but a data scrub can catch it). That's probably no so important for the relatively small array you're talking about, but the bigger you go, the more important it becomes.

Also, a RAID-5 array has one extra write per full-size stripe (yes, stripe, not strip this time) written; a RAID-10 array has an extra write per drive pair. If all of your writes on the RAID-5 were less than the full stripe size, the total number of writes would be as bad as the RAID-10, but never worse.

With the 9260, you can use SSD Guard, which catches early signs of an SSD failure, and copies all data from the failing drive to a hot standby. As Anvil suggested, that might be a good setup for a shared server: 2+1 RAID-5, plus one hot spare.

[IanB]

http://www.lsi.com/storage_home/prod...-4i/index.html

Just double checked, and even the 4i supports RAID 6. Maybe that's the compromise answer for the same or better redundancy as a RAID 10 with 4 drives. I'm not looking to search for the ultimate benchmark level of performance, the array just has to be better and more reliable than the single SAS drive it's replacing. I think that's a certainty however I configure it.

Remember I've just had a disaster situ where two drives out of a RAID 5 died... so I am keen on the redundancy aspect. I know stevecs has made strong comments about RAID 6 being superior to 5 due to the second parity check providing some data validity confirmation.

Is data scrubbing something you manually initiate, or does the controller run a scan itself on a regular or scheduled basis?

[IanB]

So... first quick tests with this new array. As I said above, I'm not interested in chasing the last extra byte of bandwidth, I'm sure there may be better configs. But I'd appreciate some experienced eyes looking at this to see if it looks about right or off.

This is 4 x Corsair Force 60GB in RAID 6 on an LSI 9260-4i without BBU, on a Gigabyte H57M-USB3 board/Core i3-530, running Win XP. The stripe is 32K, write-back on, no read ahead, direct IO. The cache on the 9260 is 512Kb. I ran the first benchmark with the setting "Disk cache" in the LSI Mega Storage Manager set to both enabled and disabled, and it made absolutely no difference to the numbers at all. There was no obvious documentation for that setting so I have no clue what that does, except (for me) not a lot.

The first bench was with random data, the second with all 1-fills, which is of course highly compressible so better on the Sandforce Corsairs. The array will be serving files that will be more compressible than not (Word docs rather than multimedia files), so I should expect to see somewhere in the middle in real usage, I guess.



I'd hoped for a little better on the 4K tests. Perhaps this is a result of the RAID6, which the LSI documentation warns would reduce writes. But I have to say that months ago I did the same benchmark on the old SAS 15K RAID5 array (on a Perc 5) this is replacing, just for a giggle, and was shocked to see 4K numbers of around or less than 1. Yes, less than 1. So I guess I just traded up to 20 or 30 times better in one go, which isn't all bad.

EDIT: just a quick addition about installation on this Gigabyte board, which may or may not be relevant for other Gigabyte boards... I am running this in a business environment, hence using the i3 inbuilt graphics facility. I therefore tried to install the 9260 in the first PCIe slot, which was free, but the board refused to boot, cycling on and off over and over, which was rather scary. Moving it to the second PCIe slot worked perfectly, though, so the H57M (and maybe other H57 boards) is obviously fussy about only allowing graphics cards in the first slot. I decided to add an extra 8cm fan at the bottom of the case directed at the heatsink, as it was very hot and I wasn't comfortable with that in a case using minimal cooling for quietness.

I was also lucky that the most recent firmware for the card, which has been giving some boards issues, worked perfectly on this too.

[Computurd]

Ian, do you have fastpath?

[IanB]

Ian, do you have fastpath?

You gotta be kidding. I can't even find where to buy a BBU for this unit normal retail in the UK, never mind fastpath dongles. I know it's nice to have, but I don't suppose it would be affordable for my school on top... unless it did really amazing things to the performance.

[Computurd]

well...thats the catch ian...it does amazing things! Especially small file random. However, not sure if your usage patterns would benefit from it..but when it comes to the small 4k files you are working with look at the difference, here is a chart, pre-fastpath v fastpath....

[IanB]

Well, alrighty then.

See, this is why I bought a 9260 knowing I had an upgrade path.

Maybe I'll treat me for Xmas if you can help me find a retailer that'll ship to the UK.

[Computurd]

well ian, they have fixed that issue. you can order online now and purchase a software key. activation is not a physical key anymore that is what this latest firmware they put up today is to address...I am putting the 9260 on right now to test...gah i cant resist!

[eva2000]

you can order online where ?

[Computurd]

I am not sure exactly what the specifics are, however, that is what this has all been heading to with the keys you can buy, they call them Advanced Services...and the whole idea is to order software keys online that they email you, so no physical key needed. if you open the "manage megaraid advanced software options" selection on the homepage of the new MSM you will see where you enter in safe id and serial numbers for these products. unfortunately it is these BS software keys that have caused the bloated ass firmwares that arent allowing these things to work with enthusiast boards. i am so pissed off i am not thinking rationally. this firmware update does nothing to fix the issues with the evga boards..
have you updated your firmware yet?
and, if so, how is it working with your asus? they have released a new FW today..and it will not work with the evga to my knowledge

[IanB]

http://www.lsi.com/channel/products/...ath/index.html

Using the "buy now" link goes straight to the online LSI store where you can get the physical dongle for $165, and it appears they ship international. Woot! I can get me a BBU for this puppy too...

That is way easier than trying to find a local reseller through their site, which doesn't even include the BIG online store I purchased this 9260 from.

[Ipreferspam]

I just flashed my 9260-4i with the latest firmware (dated 9/3/2010) and am having no issues on an Asus Rampage Formula x48. Note though that if I do not put my ICH in IDE mode and have the JMicron RAID chip on my motherboard disabled, I am unable to either install windows or get into WebBIOS.

[eva2000]

yeah only place that ships to Australia which has fastpath is provantage at US$146 but they're out of stock, so need to wait heh

[IanB]

@Eva, see two posts above yours. The LSI online store will ship internationally. Use that link to the physical fastpath dongle in their product range, then use the "Buy Now" link on the right of the page to go straight to the store listing.

[eva2000]

thanks ended up ordering via provantage.com anyway

[eva2000]

I'd hoped for a little better on the 4K tests. Perhaps this is a result of the RAID6, which the LSI documentation warns would reduce writes. But I have to say that months ago I did the same benchmark on the old SAS 15K RAID5 array (on a Perc 5) this is replacing, just for a giggle, and was shocked to see 4K numbers of around or less than 1. Yes, less than 1. So I guess I just traded up to 20 or 30 times better in one go, which isn't all bad.

yeah 4k writes scaling with sandforce ssd and 9260 don't seem to be doing much same with my 9260-8i and 8x 60gb sandforce gskill pro ssd same as with 3x 60gb sandforce in terms of performance http://www.xtremesystems.org/forums/...d.php?t=258503

[eva2000]

thanks ended up ordering via provantage.com anyway

crap so mad at the moment, provantage declined my order even when they said American Express was accepted for Australian orders