Stripe size for SSD RAID0?

[Ao1]

After reading this many times to try and figure out what is being said... I think it's trying to say that the maximum data that could be written to/from the flash media in one operation is 128K... While this is good, the best stripe size is actually more dependent on the average read/write operation that the OS requests... not on what the drive can handle. Steve said above that the OS typically reads/writes small pieces of data (as small as 8KB). If this is true, a 128K stripe will never yield any parallelism performance gains... NONE!

As I said before, benchmarks are meaningless unless they emulate the kinds of data reads/writes that transpire in typical usage. Of course a benchmark tool that reads/writes data in 512K blocks will show that a 128K stripe is the best.

I think Ourasi is on the right track.

The "benchmark" I used was a mixture of random reads and writes ranging from 16K to 1024K and the results are summarised in the excel sheet for each strip size on the linked post above. I agree with Ourasi however in that you need to find what is right for your specific set up.

EDIT:

Ourasi
Would you mind running a quick IOmeter benchmark on a 32k and 128k stripe? OCZ created a 10GB "bootup" config file to try and establish a common IOmeter test for comparative benchmark performance purposes, which is representative of what happens during a system boot. I have run this config on both ICH9r and 5405 with a 128K strip to compare the two controllers and the results were not that different. I'd be happy to convert my array to 16K on the 5405 to compare results on the stripe size on the 5405, but I'm already sure that 128k works best on my set up.
A link to the config file can be found here: http://www.ocztechnologyforum.com/fo...ad.php?t=54752

[Ao1]

IOmeter "Bootup" config file test results.

• 32k & 128k stripe results for X25-E x2 raid 0 on Adaptec 5405 controller
• 32k & 128k stripe results for X25-E x2 raid 0 on ICH9r controller

Build: QX6850 (un-clocked) 8GB OCZ DDR3 platinum edition, ATI 4870x2. W7/64.

Disks wiped with Hdderase after each benchmark.

Disks returned to 128K

[One_Hertz]

^^^^

Yep. I found the same results. 32k is slower in every way in all request sizes. That IOMeter test file tests 4kb-64kb file sizes mostly reads and some writes. This test is a strength of the 32k stripe size according to theory, but SSDs do not work like that...

You would have to either be confused or have a very specific application (i.e. server of some sort) to be running 32k on Intel drives.

I think I have seen in a few places that mtrons work best at 64kb stripes, but for all other SSDs 128kb or more is the best... The disadvantage of the 32kb stripe would be even more apparent once the drives get to the "used" state.

To all those saying that high stripe size does not help with small files - you would be right if the OS and the storage subsystem did not do any caching whatsoever, which is not true.

[NapalmV5]

theres no such thing as xxxk stripe is the best for ssd maybe in the world of intel/highpoint/adaptec

because of areca it depends on the controller..

areca: 64k no matter the # of ssds

intel/highpoint/adaptec/etc: stripe k has to be increased accordingly to the # of ssds

128k @ 2x
256k @ 4x
512k @ 8x

[NapalmV5]

@napalmv5- you hit on another aspect of this as well which makes most of the questions here hard to answer. In that raid/striping or really any api has implementation issues. Areca's perform better with certain stripe sizes in parity raids differently than non-parity, and different sizes also perform differently depending on many factors (scratch space, what type of algorithm they're using (is it 1x64, 2x32, 4x16 or whatever) and how that relates to internal bandwidth and processing power. That's where the old adage comes in 'world as designed, and world as built'.

lol im not trying to complicate things.. just pointing out the efficiency of areca controllers


even if stripe could be perfectly matched to the number of ssds @ intel/highpoint/adaptec controllers.. efficiency still dont come close to that of areca controllers

those of you that dont believe you can certainly do your own testing

[lowfat]

theres no such thing as xxxk stripe is the best for ssd maybe in the world of intel/highpoint/adaptec

because of areca it depends on the controller..

areca: 64k no matter the # of ssds

intel/highpoint/adaptec/etc: stripe k has to be increased accordingly to the # of ssds

128k @ 2x
256k @ 4x
512k @ 8x

When I had an ARC-1220 & 4 Patriot Warp V2's. I found that 128k performed the best

[annihilus]

I went raid 10 (1+0) with ich10r, wouldn't let me go higher than 64k, anybody else experience this?

areca: 64k no matter the # of ssds

I've run multiple tests with multiple ssd's and I think with raid 0 it runs better with 128k

[virtualrain]

Here are some benchmark results for a pair of Intel X25 SSD's on a Mac (software RAID0)... 3 different benchmarks... with each one seeming to favor a different stripe size.



In a few instances, Xbench seems to favor 128K stripe. Quickbench consistently scores higher with a small stripe. ZoneBench favors a 32K stripe.

[Ao1]

Here are some benchmark results for a pair of Intel X25 SSD's on a Mac (software RAID0)... 3 different benchmarks... with each one seeming to favor a different stripe size.

In a few instances, Xbench seems to favor 128K stripe. Quickbench consistently scores higher with a small stripe. ZoneBench favors a 32K stripe.

I have not heard about those benchmark programs, but if they are giving different results would it not imply that the benchmark programs are flawed?

Intel asked reviewers to format after each benchmark because the drives “adapted to use”. The “adapting to use” issue was supposed to have been fixed in the new firmware, however to take away any possibility of degraded performance I always test on a formatted drive (using hddease to properly format it).

As One_Hertz stated above a degraded drive would most likely perform even worse with a 32k stripe.

PC benchmarks for HHD do not work well with SSD, which is why the consensus seems to be that IOmeter is the only reliable way to benchmark.

The “trouble” with IOmeter is that it is not a standard test in that it can be user configured to whatever you want to test. This is not really a problem because that is exactly what it is intended to do, but obviously comparative testing will only work if you run the same test.

This is why it is quite handy that OCZ produced a standard benchmark IOmeter configuration file.

On a separate issue is the speed on a mac OS not limited by the OS itself? I recall reading something about this on the ocz forum....but I’m not a mac user so I didn’t take much notice and could be completely wrong. (Please don't shoot me for even suggesting such a thing it was just something I half read!)

[Biker]

Why did you pick 128K? Did you try other stripe sizes? What were the results?

After doing lots of testing with my LSI Megaraid 8708 custom build I found that a 128k stripe worked the best for my config (2x Samsung 64GB SLC in 2 seperate arrays) at 16, 32, 64 and 256kb stripe the random writes were dropping off from optimal.

[virtualrain]

I think there's a fundamental issue here... Do any of you agree?

The issue may be that most synthetic benchmarks favor one stripe size over others due to the nature of their design. Therefore, if you use synthetic benchmarks to determine your ideal stripe size, you may be compromising performance under real-world conditions. For example, just because benchmark XYZ shows the best performance with a 128K stripe does NOT mean that's going to yield the best real-world storage performance right? Yes/no?

So I go back to my original assertion... forgetting benchmarks for a moment... doesn't it make sense that to maximize performance, you want even the smallest read/write to be striped across multiple drives? Therefore the smaller the stripe, the better the I/O performance for all file sizes?

Don't you guys realize that by using a stripe of 128K that you are effectively eliminating any benefits of striping for small files?... Do you not think this is important? Are all your files greater than 128K?

Things are still not making sense to me.

[Ao1]

^^
I have not been able to find a professional review that investigates raid stripe sizes with SSD. I’ve found reviews of sdd in raid but most don’t bother to give details of what stripe size they used when testing SSD in raid 0 and why they used it, in fact I have only been able to find two test benchmarks that specify which stripe was used and both coincidently used 64k.

Areca used a 64k stripe for benchmarking 5 X-25-e’s on the ARC-1231ML. Results here (check out the pdf download in relative resource box)

AnandTech used a 64k stripe to test up to 8 X-25-e’s. Results here

All I can say is that on my system anything less than 64k does not work as well in various synthetic testing, but of course it’s all about what feels right and works for your particular set up and use.

The reason hdd benchmarks can give wrong results for sdd is partly explained here

[One_Hertz]

Don't you guys realize that by using a stripe of 128K that you are effectively eliminating any benefits of striping for small files?

No, the storage subsystem is not that dumb, so to speak. When it detects you are trying to sequentially access a some small files it predicts you will keep accessing and loads megabytes of those files from the array into cache first (read ahead). It is essentially guessing that if you have accessed files 1,2,3 that you will continue accessing 4,5,6,7,etc. and loads all of those as a chunk into the cache. This chunk is going to be a larger piece of data which will stripe between different drives. That's why we see ~400MB/s from two X25-Es when accessing 4kb-64kb files on 128kb stripe. If what you were saying was true (and it is true if there was no cache) then we would not get performance that is higher than a single drive (2xxMB/s).

I am probably not 100% correct and accurate with my explanation, but that is the general idea. Perhaps stevecs or someone else can explain it better.

[virtualrain]

No, the storage subsystem is not that dumb, so to speak. When it detects you are trying to sequentially access a some small files it predicts you will keep accessing and loads megabytes of those files from the array into cache first (read ahead). It is essentially guessing that if you have accessed files 1,2,3 that you will continue accessing 4,5,6,7,etc. and loads all of those as a chunk into the cache. This chunk is going to be a larger piece of data which will stripe between different drives. That's why we see ~400MB/s from two X25-Es when accessing 4kb-64kb files on 128kb stripe. If what you were saying was true (and it is true if there was no cache) then we would not get performance that is higher than a single drive (2xxMB/s).

I am probably not 100% correct and accurate with my explanation, but that is the general idea. Perhaps stevecs or someone else can explain it better.

That makes sense of course. Cache is huge factor in improving performance. However, aren't you indirectly saying that the cache effectively compensates for a poor choice of stripe size?

I still think that with a 128K stripe, you aren't getting much benefit by running your drives in RAID0... you would be better off running JBOD with the OS on one drive and apps on the other... at least that way you gain some parallelism from reading different kinds of files.

I find it ironic that many people are extremely concerned with small random read/write performance with SSD's but then put them in RAID0 with a 128K or 64K stripe? This will not improve small random read/write performance. It will help sustained transfer rates of very large files, but that's not where SSD's need help.

[One_Hertz]

That makes sense of course. Cache is huge factor in improving performance. However, aren't you indirectly saying that the cache effectively compensates for a poor choice of stripe size?

I still think that with a 128K stripe, you aren't getting much benefit by running your drives in RAID0... you would be better off running JBOD with the OS on one drive and apps on the other... at least that way you gain some parallelism from reading different kinds of files.

I find it ironic that many people are extremely concerned with small random read/write performance with SSD's but then put them in RAID0 with a 128K or 64K stripe? This will not improve small random read/write performance. It will help sustained transfer rates of very large files, but that's not where SSD's need help.

The cache does cover for the small files, but what makes large stripes a better choice is that there is much less overhead and the small random reads/writes are going to be slightly better for larger stripes.

Having smaller stripes will make it so that more random requests hit on the stripe boundaries. When that happens you have to wait for both drives to do the access for the operation to be complete instead of just waiting for one. When doing small reads the seek time is going to be much greater than the actual read time so that is what you care about. Seek time of one drive on average will be better than the worst seek time taken from two drive.

Small stripe sizes also increase write amplification and do a bunch of nasty things to writes but I wrote about this somewhere already and don't want to rewrite.

And yes, if you separate two drives and actually know that your requests will fall on both drives, that setup will be much better than raid 0. It is just that most of the times you are going to care about the performance of a single app the most, in which cases this won't help.

[F.E.A.R.]

@ audienceofone

Upload the Bootup-Pattern, please

[virtualrain]

Having smaller stripes will make it so that more random requests hit on the stripe boundaries. When that happens you have to wait for both drives to do the access for the operation to be complete instead of just waiting for one. When doing small reads the seek time is going to be much greater than the actual read time so that is what you care about. Seek time of one drive on average will be better than the worst seek time taken from two drive.

Small stripe sizes also increase write amplification and do a bunch of nasty things to writes but I wrote about this somewhere already and don't want to rewrite.

This is one of the main problems here... we shouldn't use old-school theory for magnetic media to define the ideal SSD stripe size. Remember, the 64K stripe size was born out of a need to balance seek times on magnetic disks with increased parallelism... (as you point out)... it was a compromise dictated by the constraints of the devices being used. A 128K stripe traded more parallelism for faster access times which benefited large files (e.g. movies and other multi-media objects). But at any rate, the theory behind these large stripe sizes was based on the limitations of magnetic media.

And yes, if you separate two drives and actually know that your requests will fall on both drives, that setup will be much better than raid 0. It is just that most of the times you are going to care about the performance of a single app the most, in which cases this won't help.

Not necessarily... most applications rely heavily on OS or common API's that will be fetched from your OS drive so any app will gain added parallelism using this kind of storage model.

At any rate, my suggestion to use JBOD disks was merely an extension of using ridiculously large stripe sizes. At 128K stripe sizes, there's very little benefit to be gained from RAID0 vs. JBOD with typical disk I/O.

[Ao1]

^^
Virtualrain, you asked what people used and if choice was based on evaluation. I’ve tried to show you the evaluation I undertook to come to my conclusion. Do you think my evaluation methods were flawed? Why do you think my IOmeter results were worse with a 32k stripe? When I ran random read & writes I only got 5.24MB/s 4k writes with a 16k stripe as opposed to 199.38 with a 128k stripe.

If I’m missing a trick I’d like to know. If you think my evaluation methods are wrong please explain why and what you think would be a better way of testing. I can reset my stripe size on line so I’d be happy to run whatever test you think would be relevant.

[virtualrain]

^^
Virtualrain, you asked what people used and if choice was based on evaluation. I’ve tried to show you the evaluation I undertook to come to my conclusion. Do you think my evaluation methods were flawed? Why do you think my IOmeter results were worse with a 32k stripe? When I ran random read & writes I only got 5.24MB/s 4k writes with a 16k stripe as opposed to 199.38 with a 128k stripe.

If I’m missing a trick I’d like to know. If you think my evaluation methods are wrong please explain why and what you think would be a better way of testing. I can reset my stripe size on line so I’d be happy to run whatever test you think would be relevant.

I really don't know what to think. This thread is a mixed bag of theory (some of which seems old-school and/or irrelevant), evaluation techniques (if any) and varying results (I personally found three different benchmarks that each favored a different stripe size).

Your results seem incredibly bizarre. Does it make sense to you that stripe size has a 40x impact on I/O performance? While I don't doubt your results, there must be something we don't understand here or that benchmark is seriously flawed in this application.

[One_Hertz]

I really don't know what to think. This thread is a mixed bag of theory (some of which seems old-school and/or irrelevant), evaluation techniques (if any) and varying results (I personally found three different benchmarks that each favored a different stripe size).

Your results seem incredibly bizarre. Does it make sense to you that stripe size has a 40x impact on I/O performance? While I don't doubt your results, there must be something we don't understand here or that benchmark is seriously flawed in this application.

I don't see why you are confused.

Sequential small files = both stripes should give similar performance due to caching. 128 is preferred due to less overhead.
Sequentual large files = 128 is better.
Randoms = 128 is better.

What else is there to talk about?

[virtualrain]

After a lot of work testing 16kb, 32kb, 64kb and 128kb stripes, I found 16kb and 32kb to be the best, and 64kb just a bit behind, and 128kb to be pretty far behind, in benches/stopwatch in apps/tools/utilities I consider to be important for OS operation and work, and also game/map loading.
In terms of IOPS on random read/write 4kb and other small file sizes they where all equal.

At the moment I'm on 16kb stripe, and have been for a while now, and it is blisteringly fast in real world, not only in benches. There is not much performance difference between 16kb - 32kb - 64kb, but the smaller sizes edges ahead. As long as there is no measurable negative impact using these small stripes, I will continue using them. The X25-M is a perfect companion for those small stripes, some SSD's might not be. Intel was right about this in my opinion. But I must add: This is with my ICH9R, and I fully understand that these stripes might perform pretty bad on some controllers, and that's why I encourage people to test for them selves...

I think it's worth re-iterating this post. Without consensus, I'd stay there's still plenty to discuss and understand.

[One_Hertz]

I think it's worth re-iterating this post. Without consensus, I'd stay there's still plenty to discuss and understand.

There are like 10 times more people saying that their testing showed 128 or more to be the best. Who do you think tested properly? Things like this will never be 100% one sided.

[virtualrain]

There are like 10 times more people saying that their testing showed 128 or more to be the best. Who do you think tested properly? Things like this will never be 100% one sided.

Fair enough.

Here's an insightful post from another forum which attempts to explain why 128K is working better...

Most SSD's have a native block size of 32KB when erasing. Some have 64KB. This is the space that has to be trimmed or "re-zeroed" before a write. If you write 1,024KB to a RAID-0 pair with 64KB blocks, with a 32KB stripe size, it will be 32 writes, requiring 64 erases. With 128KB stripes, it will be 32 writes, 32 erases. You'll effectively see a 30-50% difference in writes. This does not affect reads quite as much, but it's still usually double-digits. Also, with double the erase cycles, you will cut the lifespan of the drive in half.

With small writes, who cares? It takes you how long to write 4K at 200MB/s? When it really matters is when you load that 500MB file, or a 300MB game executable plus 1100MB of textures.

Something that can affect speed as much, if not more than stripe size is array offset(aka alignment). Starting on an even drive block, and lining up your sectors to that block can make a huge difference. If you're unsure, just go with 256KB or even 1024KB offset, and you will almost certainly land on the beginning byte of a whole block, which will fit fine to nearly any Windows OS's default partition sector size.

[Ao1]

As far as I know Intel have a 128k erase block size, however Intel also state that they have figured out a way to program and erase just what is necessary (in small bytes rather than large blocks).

As far as I know the Vertex uses a 512K block size and the complete block gets erased.......but I’m not 100% sure of that.

The lack of clarity seems to be that no one wants to reveal too much about the inner workings of their controllers, which really makes this issue quite hard to understand.

I suspect other issues like write combining also play into this, but again how this works is not really known.

I think all we can be sure about is that a stripe size under 64k is not doing anyone any favours, but as you say there's still plenty to discuss and understand.

[F.E.A.R.]

@ audienceofone

Upload the Bootup-Pattern, please

*cough*


@ all

I´m using 128k stripesize