LSI 9260 vs. AMD SB850

Printable View

12-23-2010, 04:40 AM
F@32

LSI 9260 vs. AMD SB850

Hi all,

Today I embark on a quest to figure out if RAID controller which is three times as expensive as the motherboard it will be plugged into, can outperform said motherboard in simple 2-SSD drive RAID-0 configuration. I solicit advice from all the storage gurus on how to tweak LSI controller for the optimal performance. As you can see LSI is falling short in AS SSD benchmark. Any thoughts?

The set-up:
Drives: Intel X18-M Gen1 50nm 80Gb, 4K NTFS allocation unit
Controller: LSI 9260-4i, FW 12.11.0-0016, Driver 4.32.0.64, no FastPath
Mobo: Gigabyte GA-890GPA-UD3H, bios F8, 8.71 storage driver
CPU: Phenom II 965, TRUE-120 default clocks
RAM: 2x2Gb DDR3 G-Skill Ripjaws F3-12800cl9d
Video: Asus HD5850, Cat 10.6
Sound: M-Audio Audiophile Delta 2496
NIC: Intel Gigabit CT PCIe
PSU: Antec Quattro 850Wt
Case: Antec 1200
OS: Windows 7 Ultimate 64bit

LSI 9260 settings will be listed in the following order:

Code:

Stripe - 128K Read Policy - ARA Write Policy - AWB IO Policy - Cached IO Access Policy - Read Write Disk Cache Policy - Enabled BAckground Initialization - Enabled Windows Write-caching policy Write caching - Enable Write-cache buffer flushing - Disable

1. SB850 Windows 7 soft raid
http://i15.photobucket.com/albums/a3...50Softraid.png

2. SB850 128K stripe, NCQ on, write-back on, flushing off
http://i15.photobucket.com/albums/a3...Hybridraid.png

3. LSI 128K stripe (128 ARA AWB CIO RW E E. E D)
http://i15.photobucket.com/albums/a3...BCIORWEEED.png

4. LSI 16K stripe (16 ARA AWB CIO RW E E. E D)
http://i15.photobucket.com/albums/a3...BCIORWEEED.png

5. LSI max throughput check
http://i15.photobucket.com/albums/a3...Throughput.png
12-23-2010, 05:16 AM
alfaunits

Since when is 9260 that cheap to cost only 2 AMD mobos?
12-23-2010, 06:22 AM
F@32

Quote:

Originally Posted by alfaunits

Since when is 9260 that cheap to cost only 2 AMD mobos?

Lol, good catch, fixed it ;-)
12-23-2010, 10:46 PM
Computurd

do you have fastpath?
12-24-2010, 02:39 AM
Anvil

3 Attachment(s)

I just performed a few tests on my 2R0 G2 160GB on my 9260 w/FP key and the latest LSI firmware..
(the drives are in steady-state)

Strip size 32KB, NRA (no read ahead), DIO (direct IO), WB (write back)

I was a bit surprised to see that Seq. read on AS SSD is quite low, have to check that one with previos tests.

Your results do look low, do not use Read Ahead on SSDs using the 9260, I would also try changing from cached IO to Direct IO.
12-24-2010, 05:08 AM
F@32

Quote:

Originally Posted by Anvil

I just performed a few tests on my 2R0 G2 160GB on my 9260 w/FP key and the latest LSI firmware..
(the drives are in steady-state)

Strip size 32KB, NRA (no read ahead), DIO (direct IO), WB (write back)

I was a bit surprised to see that Seq. read on AS SSD is quite low, have to check that one with previos tests.

Your results do look low, do not use Read Ahead on SSDs using the 9260, I would also try changing from cached IO to Direct IO.

Thanks Anvil, will fire her up :D Yep, something is fishy with AS SSD. Works fine with onboard...
CT, no FP at the moment.
12-24-2010, 01:17 PM
odditory

I don't trust any results from AS SSD. From the first time I ran it, I knew it was screwy.
12-24-2010, 03:59 PM
F@32

Anvil, note mine are 80Gb G1, so they are 70Mb/s on writes... I knew that Id need bunch of drives to push 9260, and it looks nice with 4, but not sure I like $/performance especially with added read latency. Note the latency with re-badged A-Data SF1222 drives. Wow...

OK, here's 2R0 32K, DIO, NRA, AWB
http://i15.photobucket.com/albums/a3...BDIORWEEED.png

2R0, 128K, DIO, NRA, AWB
http://i15.photobucket.com/albums/a3...BDIORWEEED.png

3R0 with SF1222 drives (not impressed with them), 128K, ARA, AWB, CIO
http://i15.photobucket.com/albums/a3...BCIORWEEED.png

4R0 16K (2 X18M, 2 SF 1222)
http://i15.photobucket.com/albums/a3...BCIORWEEED.png

4R0 128K (2 X18M, 2 SF 1222)
http://i15.photobucket.com/albums/a3...BCIORWEEED.png

May be AMD mobo doesn't like it?
12-25-2010, 11:04 AM
Anvil

3 Attachment(s)

F@32,
Could be the AMD/LSI combo, I think it might be worth getting the FP key, as you can see from my tests it scales fairly OK.

4R0 Vertex 2 on the 9260 w/FP key.

Personally I don't have any issues with the SF based SSDs :)

AS SSD
Attachment 110604

CDM using 0-Fill option (best case)
Attachment 110605

random data (worst case)
Attachment 110606
12-25-2010, 12:07 PM
F@32

Thanks Anvil. FP would be sweet, but that would mean another $150. Thing is SB850 with one worker @ QD16 gives me 50K IOPS with 4K random. 9260 without FP at least on my PC is 35K... That's not right ;)

http://i15.photobucket.com/albums/a3...0QD167-4-2.png
12-25-2010, 12:20 PM
Computurd

FastPath will do 160,000 IOPS @4k random.
12-25-2010, 12:23 PM
Anvil

I'll see if I can do a test of the 9260 on my AMD rig sometime soon, the results should compare to my latest results on the Intel setup.
(might be able to squeeze it in tomorrow)

The 9260 has always been a screamer at high QD, the FP key somewhat changed that.

Do you need >50K iops or is it just for the comparison?
12-25-2010, 12:29 PM
Anvil

2 Attachment(s)

Quote:

Originally Posted by Computurd

FastPath will do 160,000 IOPS @4k random.

Somewhat close to 150-160' iops is correct, depends on the SSDs in question.

This is the 4R0 Vertex 2 using a 5GB testfile.

Attachment 110608
12-25-2010, 02:01 PM
F@32

Quote:

Originally Posted by Anvil

Do you need >50K iops or is it just for the comparison?

Just for comparison, "free" fake raid shouldn't be outperforming 9260 :)
12-25-2010, 02:32 PM
Computurd

Quote:

"free" fake raid shouldn't be outperforming 9260

with only two devices it might outperform it in some scenarios. but not in stability of course. a few bluescreens and your array is a goner on most fakeraids.
however, you shouldnt be buying a 500 raid card for two devices anyway. this is for larger arrays, where the ICH or SB850 couldn't even begin to keep up.
also, i couldnt help but notice that you are comparing a soft raid to a real raid. any type of software raid will be faster, while also being inherently unstable, and chew your cpu resources up as well.
that doesnt seem to be a fair representation. the 9260 is designed for uber throughput, with large arrays for specific use in enterprise environments, and supercomputing. it is tuned for large arrays and scales beautifully with them.
for low latency usage they have the fastpath key, which will tear up anything that you put it up against.

Quote:

Note the latency with re-badged A-Data SF1222 drives.

havent heard good things about that set of drives. seems to be knock-offish with poor firmware support.
12-25-2010, 02:57 PM
Anvil

There is more to it than just the iops.

@50K iops you are consuming > 1/3 of the CPU on the AMD rig, the 9260 has it's own "cpu" and shouldn't be consuming > 10% in most cases.
As the QD increases, CPU utilization on the AMD (and Intel's as well) steadily increases and in the end they'll consume most of the processing power available on the computer, this comes at a cost. (other tasks suffer)

OTOH, there was never any doubt about the onboard controller being competitive for most desktop tasks, just look at the ICH, a race-horse at heart and the SB850 is indeed a great step up from previous chipsets.
12-25-2010, 03:01 PM
F@32

Quote:

Originally Posted by Computurd

with only two devices it might outperform it in some scenarios. but not in stability of course. a few bluescreens and your array is a goner on most fakeraids.
however, you shouldnt be buying a 500 raid card for two devices anyway. this is for larger arrays, where the ICH or SB850 couldn't even begin to keep up.

Oh, I totally agree. I wouldn't have paid full retail for it, temptation was great to mess with it. I will be frank, I feel that FP idea is a rip off. Should have been enabled by default.

Quote:

Originally Posted by Computurd

havent heard good things about that set of drives. seems to be knock-offish with poor firmware support.

They are back to where they came from. But FW seemed to be 3.1.2 SF1222...

Quote:

Originally Posted by Anvil

@50K iops you are consuming > 1/3 of the CPU on the AMD rig, the 9260 has it's own "cpu" and shouldn't be consuming > 10% in most cases.

Yep, you're right there. Noe that in the test only 1 worker was running. For desktop/gaming use I sure hope 4K >QD16 is not a very common scenario. In any case, at least there is some utilization for multiple cores.

6% ;)
http://i15.photobucket.com/albums/a3...BDIORWEEED.png

And yes, at QD32 with 4 workers AMD is tapped out. You have to agree I hope, that 40% CPU sacrifice for 60K is worthy.

http://i15.photobucket.com/albums/a3...KRRQD321WK.png

http://i15.photobucket.com/albums/a3...RRQD164WKS.png

http://i15.photobucket.com/albums/a3...RRQD324WKS.png

4R0 LSI no FP (again, something is borked on that mobo)
http://i15.photobucket.com/albums/a3...BCIORWEEED.png

4R0 AMD
http://i15.photobucket.com/albums/a3...X18MSF1222.png
12-26-2010, 08:36 PM
Computurd

those are some nice numbers, however, the fact that you cant boot a soft array just totally ruins it for me. i just have no use for a non-boot array personally. even though i have toyed with some passthrough soft array on the 9211 with five devices and five on a 9260 as the boot. uber fast to say the least :)

tbh that cpu usage you are showing with that is shocking! wow 78 percent! ive never seen ICH kick off utilization like that with a soft array, however, i havent toyed with it too much....but that number seems abnormally high to me.

does your mobo have the NF200? seems like some motherboards and the nf200 do not mix AT ALL with the 9260...like kryptonite on the classified e759 :(
12-27-2010, 03:28 AM
Anvil

F@32

I had a shot at the 9260 on my Asus CHIV, it didn't work as my boot drive was configured for RAID and and as soon as I installed the 9260 it just wouldn't let me select my boot drive again.
All that was left was the 9260 and I can't reconfigure the drives right now.

Will give it a go at some later point in time.

The performance you saw might just be a result of mixing drives, sometimes it works OK but not all of the time.
After all the X25 and the SF are different beasts.
12-27-2010, 04:10 AM
alfaunits

Quote:

Originally Posted by F@32

And yes, at QD32 with 4 workers AMD is tapped out. You have to agree I hope, that 40% CPU sacrifice for 60K is worthy.

Not really, but I am quite happy to see AMD's drivers actuall use multiple cores! LSI 9211 would only use ONE core on my C2Q build, and it was obvious it is exactly CPU limited. (it gave awesome numbers though :P)
I think that also Intel's ICHxR will not use more than one core. Windows "RAID" won't use a full core even on a large RAID5, I don't know why :( (neither will it use it for EFS!). I am at a loss why.
12-27-2010, 08:26 AM
Anvil

To achieve ~100' IOPS on the SB850 I had to use 4 or 6 workers (iometer), CPU utilization >90% :)

Jasper Forest might change raid-5 performance on the Intel platform :)
12-27-2010, 08:36 AM
F@32

Quote:

Originally Posted by Computurd

those are some nice numbers, however, the fact that you cant boot a soft array just totally ruins it for me. i just have no use for a non-boot array personally.

Hey CT, sorry if I was not clear: those numbers are from SB850 RAID which is bootable. I only did pass through mode once to see if there was anything funny going on.

Quote:

Originally Posted by Computurd

tbh that cpu usage you are showing with that is shocking! wow 78 percent! ive never seen ICH kick off utilization like that with a soft array, however, i havent toyed with it too much....but that number seems abnormally high to me.

Yep, I'm glad AMD drivers utilize more than one core even with 1 worker.

Quote:

Originally Posted by Computurd

does your mobo have the NF200? seems like some motherboards and the nf200 do not mix AT ALL with the 9260...like kryptonite on the classified e759 :(

Nope, that's a 890GX/SB850 mobo.

Quote:

Originally Posted by Anvil

F@32

I had a shot at the 9260 on my Asus CHIV, it didn't work as my boot drive was configured for RAID and and as soon as I installed the 9260 it just wouldn't let me select my boot drive again.
All that was left was the 9260 and I can't reconfigure the drives right now.

Will give it a go at some later point in time.

Appreciate your help. I would like to see someone else's AMD+LSI numbers. I experience problem installing Win7 on LSI array. Every other or so re-boot it drops one or both drives. No problems otherwise... Right now I'm not too happy with LSI 4K IOPS without FP. 40% CPU utilization with AMD QD16 and 50K IOPS vs 6% utilization with LSI and 35K IOPS... For $350 card I would have expected it to whipe the floor with SB850 even with 2R0, but it seems another $150 needed to match and beat it. That's $500 vs something that's already there. This build is for gaming and 24/7 use. I don't think I will see benefits going 4R0, bottleneck is be back on CPU anyway.

Quote:

Originally Posted by alfaunits

Not really, but I am quite happy to see AMD's drivers actuall use multiple cores! LSI 9211 would only use ONE core on my C2Q build, and it was obvious it is exactly CPU limited. (it gave awesome numbers though :P)
I think that also Intel's ICHxR will not use more than one core. Windows "RAID" won't use a full core even on a large RAID5, I don't know why :( (neither will it use it for EFS!). I am at a loss why.

Yeah, 9211 is a darling but LSI shows no love with gimped firmware and drivers. I could imagine with proper multi-core support and above 128K sequential scaling it could fly... I'm still thinking about picking up IBM M1015 re-badged LSI 9240 for the heck of it... Funny thing is, it seems 9240 is not gimped like 9211. And it's only $20 or so more expensive.

Quote:

Originally Posted by Anvil

To achieve ~100' IOPS on the SB850 I had to use 4 or 6 workers (iometer), CPU utilization >90% :)
Jasper Forest might change raid-5 performance on the Intel platform :)

Nice, was it in 4R0 SF?
12-27-2010, 09:26 AM
Anvil

Quote:

Originally Posted by F@32

Nice, was it in 4R0 SF?

Not sure, could have been the Intel G1's, I'll check.
12-27-2010, 10:37 AM
F@32

OK, 9240-8i aka 9220-8i aka M1015 is on its way. Will see what it can do.
12-27-2010, 12:11 PM
mattkosem

Those numbers on SB850 look a bit low. I don't know if it's the drives not playing nice, or some other factor. But I generally see >1GB/sec reads from SB850. http://www.xtremesystems.org/forums/...6&postcount=93

Quote:

Originally Posted by Computurd

the fact that you cant boot a soft array just totally ruins it for me.

Seriously. Microsoft needs to step up with that. OSX can boot from soft arrays without issues, even on a hackintosh.

--Matt
12-27-2010, 12:40 PM
Anvil

1GB/s is easily achieved using 6Gb/s SSDs, F@32 was using 3Gb/s SSDs.
12-27-2010, 02:01 PM
F@32

Quote:

Originally Posted by mattkosem

Those numbers on SB850 look a bit low. I don't know if it's the drives not playing nice, or some other factor. But I generally see >1GB/sec reads from SB850. http://www.xtremesystems.org/forums/...6&postcount=93
--Matt

Lol, my sequential are low, your randoms are low... Have you sorted that out yet? And yeah, I used 5R0 SATA2 drives which in theory should have pushed over 1Gb/s sequential read (3*200+2*250).
12-27-2010, 08:18 PM
Computurd

Quote:

LSI 9211 would only use ONE core on my C2Q build, and it was obvious it is exactly CPU limited. (it gave awesome numbers though :P)

i wonder if you could isolate that process and/or driver and force affinity and the priority via the task manager? would be interesting to try....

cant wait to see your results with the new card :)
12-27-2010, 08:40 PM
alfaunits

Quote:

Originally Posted by Computurd

i wonder if you could isolate that process and/or driver and force affinity and the priority via the task manager? would be interesting to try....

It is probably not possible the way the driver is written. If it has a global lock of some sort, it would be impossible to force multiple threads. (i.e. it could be doing the processing in its own system thread, a single one - nothing one can do here).
12-27-2010, 08:48 PM
F@32

Quote:

Originally Posted by Computurd

cant wait to see your results with the new card :)

Looks like MSM and drivers can be loaded from LSI. Not sure I will want to risk FW as well right away ;) If it can match SB850 4K IOPS and show only 10% on CPUs - I will have desktop winner for 2R0/4R0.
12-28-2010, 02:12 AM
Ao1

1 Attachment(s)

So without the fastpath key a single SSD can outperform an array when it comes to 4K random reads at QD1.

I think it really sucks when hardware is deliberately gimped with software. :down:

It’s kind of strange that ICH performs so well with SSD as it predates mainstream SSD technology and could not therefore have been specifically designed to perform so well with SSD in mind. I bet Intel are kicking themselves.
01-01-2011, 08:42 PM
mattkosem

1 Attachment(s)

Quote:

Originally Posted by F@32

Lol, my sequential are low, your randoms are low... Have you sorted that out yet? And yeah, I used 5R0 SATA2 drives which in theory should have pushed over 1Gb/s sequential read (3*200+2*250).

More or less, yes. The latest bios and drivers picked up the pep on pretty much all of the randoms, but at the cost of deep queue random read perf. The seq reads vary by 100MB/sec or so from test to test, so I wouldn't read too far into it.

--Matt
01-01-2011, 11:39 PM
F@32

Quote:

Originally Posted by mattkosem

More or less, yes. The latest bios and drivers picked up the pep on pretty much all of the randoms, but at the cost of deep queue random read perf. The seq reads vary by 100MB/sec or so from test to test, so I wouldn't read too far into it.

--Matt

Nice, those are some good numbers. So, it looks like SB850 is capped at 1Gb/s or so, right?
01-02-2011, 07:27 PM
mattkosem

Quote:

Originally Posted by F@32

Nice, those are some good numbers. So, it looks like SB850 is capped at 1Gb/s or so, right?

Yeah, roughly. The highest I've seen in any benchmark was 1149MB/sec on winsat. Others seem to top out around 1050 or so though. It looks like 1037MB/sec is the highest I've seen on it with CDM or AS SSD.

--Matt
01-22-2011, 08:22 AM
F@32

So, I'm not alone to experience sub-par 9260 performance without FastPath...

The thread: 3R0 Vertex LE

And the results from the thread:
http://public.bay.livefilestore.com/...0AM.png?psid=1
01-22-2011, 02:04 PM
Computurd

run without always read ahead, and turn disk cache policy to direct. that will fix it somewhat :) at least thats where the guy linked went wrong.
what settings are you using?
01-22-2011, 02:18 PM
felix_w

F@32 , i noticed that your comparisons involve 1000MB for LSI and 100MB for SB850 when running CDM.

Could that be a possible cause for inconsistency in performance comparison ?

I also have this LSI card and a pile of SB600, SB710, SB750 mainboards if i could help with any testing, i'll be glad to do it....

...though only tested with my SB750 ASUS, so i don't know if it really is compatible with the others.

EDIT : Here is my R-0 with 4x Vertex 30GB (FW1.5) - NoRA-WBC-DirectIO on LSI (fw i think is the first released one)
12h Bios Screen

http://i256.photobucket.com/albums/h...in7-2ndBen.png
01-22-2011, 02:19 PM
F@32

Quote:

Originally Posted by Computurd

run without always read ahead, and turn disk cache policy to direct. that will fix it somewhat :) at least thats where the guy linked went wrong.
what settings are you using?

That's his results, and from what I read in that thread, once he got FP and set everything as you recommended, it was all nice and smooth.

I returned my 9260 since without FP I didn't think its 2R0 performance justifies price premium over SB850 or PCH no matter which settings I used. Of course what ASSSD and CDM don't tell you is that 4K QD32 will tax CPU at over 30%, while 9260 keeps it under 10%.

http://i15.photobucket.com/albums/a3...Cachetests.png

Quote:

Originally Posted by felix_w

F@32 , i noticed that your comparisons involve 1000MB for LSI and 100MB for SB850 when running CDM.

Could that be a possible cause for inconsistency in performance comparison ?

Possible, but not very likely. Drives were HDDErased before each HW change. And 9260 needs 1Gb test run, otherwise I'd have been benching cache (512MB on-board). The results were not totally surprising, since it's hard to beat ICH or PCH with 2R0. LSI is meant for massive throughput with 4+ devices. For OS/apps purposes I plan to stay on 2R0, and I was looking if there was anything faster while not costing a grand.

See below (I believe all were done with 64K strip):
http://i15.photobucket.com/albums/a3...v9260v9240.png
01-22-2011, 02:39 PM
Computurd

Quote:

F@32 , i noticed that your comparisons involve 1000MB for LSI and 100MB for SB850 when running CDM

Lol, how in the world did i miss this???

jesus, that makes a WORLD of difference!
01-22-2011, 02:46 PM
felix_w

Quote:

Originally Posted by Computurd

Lol, how in the world did i miss this???

jesus, that makes a WORLD of difference!

:rofl:

...been doin' a lot of practice lately in the "Find the differences" game....:p:
01-22-2011, 02:53 PM
F@32

Quote:

Originally Posted by Computurd

Lol, how in the world did i miss this???

jesus, that makes a WORLD of difference!

I see that 9260 with 100Mb test would show close to 1.5Gb/s cache performance. But how does it impact cacheless runs? I will run P67 with 1Gb and 100Mb to see if it matters.
01-22-2011, 03:08 PM
felix_w

Another thing we could see is the performance of SB850 @ CDM for 1000MB test, maybe it shows interesting things.

If you have time and the setup is ready, it would be great.
01-22-2011, 03:24 PM
F@32

EDIT: I noticed that LSI chose to gimp 4K performance of cards without FastPath. Looking at Felix's 9260, that chap on OCZ forums and mine 9260/9240 it is very evident. Now, we know that FastPath unleashes the beast, but vanilla 9260 not being able to beat SB850 in QD1-32 4K writes? It's a joke. So, not gimped 9260 is actually $500 part, not $350...

Quote:

Originally Posted by felix_w

Another thing we could see is the performance of SB850 @ CDM for 1000MB test, maybe it shows interesting things.

If you have time and the setup is ready, it would be great.

Felix, sorry my SB850 was sold. I only have P67 right now and LSI 9240 laying around. Anyway, here's back to back CDM on P67 with 100Mb and 1Gb test runs. Note that this is dirty drive with OS installed even though test was performed while I booted from another partition. As I suspected it doesn't matter much for cacheless runs. I can show AS SSD scores they also point that on-board RAID is faster. But AS SSD shows really low LSI sequential reads, so I don't know if it can be trusted there...

http://i15.photobucket.com/albums/a3...100Mbvs1Gb.png

And that's of course how 9260 looks like when testing cache vs. drives ;)

http://i15.photobucket.com/albums/a3...M50MBvs1Gb.png