Intel X25 V x 3 Raid 0 results

[ga1ve1an]

First thanx GullLars for the advice from Anandtechs review. Ran some real world type tests on 16k (stated to be recommended for intel ssd's) and 128k (some swear by this one) stripe sizes. The 16k is definitely faster than the 128k (at least for intel) and i havent even fully optimized the OS yet. The 16k is also more responsive to me as Ourasi had mentioned in one of his posts. Here is a screenshot below. Left side is 16k stripe and right side is 128k stripe. Stuck to benchmarks that represent as much real world type stuff as possible since I dont like the synthetic type stuff. Pcmark05 and Vantage HDD Tests. But threw in a CrystalMark run also.



Also noticed this under Intels control center..



So it does seem that intel recommends the 16k also!! Also noticed that the Intel SSD's do clean themselves pretty well in Raid 0. Noticed that during testing, since subsequent tests would show lower results. Just had to wait a bit betweeen each run to get a realistic picture. Dont know how much trim would improve performance since the controller does a pretty good job itself it seems.

P.S. Tests done on an ICH10R

[GullLars]

What drivers are you using?
I've seen noticably higher PCMV HDD scores from 3R0 x25-V. Another guy here on the forum got 74K, you got 51K.

BTW, impressive 4KB random write in crystal there, 125MB/s @ QD 1. 71K random read IOPS (285MB/s) is not bad either.

[ga1ve1an]

Hey GullLars..

Using Intel's most recent RST drivers. Only thing i hadnt done on this test setup is turn off Superfetch and Prefetcher in Windows 7. You dont think that would have made much of a difference, do you? I dont have a baseline to compare another with so dont know where there may be performance problems. Let me know what else you may see that doesnt seem right.

[Musho]

What about using a 4kb stripe size? Would that yield a better HDD score in Vantage?

[Musho]

Bump! Have stripe sizes of 8kb and 4kb been tested in vantage to see if the score increased even more over 16kb?

[Computurd]

only problem with the 16k stripe is that it is going to wear on your drives faster.

[ga1ve1an]

GullLars already confirmed that is not true based on how the Intel drives work. They Write Combine in blocks everything together instead of writing every little file each time which would wear faster(also known as write amplification). He commented on that on my original post at Anandtechs Raid Review here...

http://www.anandtech.com/Comments/sh...lComments=true

His quote

"There is no impact on write amplification from the striping. It can be explained pretty simply.
SSDs can only write a page at a time (though some can do a partial page write), and page size is 4KB on current SSDs. As long as the stripe size is above 4KB, you don't have writes that leaves space unusable.
With a 16KB stripe size, you will write 4 pages on each SSD alternating and in sequential LBAs, so it's like writing sequential 16KB blocks on all SSDs, and as the file size becomes larger than {stripe size}*{# of SSDs} you will start increasing the Queue Depth, but it's still sequential.

Since all newer SSDs use dynamic wear leveling with LBA->physical block abstraction, you won't run into problems with trying to overwrite valid data if there are free pages.

The positive side of using a small stripe is a good boost in files/blocks between the stripe size and about 1MB. You will see this very clearly in ATTO as the read and write speeds doubles (or more) when you pass the stripe size. F.ex. 2R0 x25-M with 16KB stripe jumps from ~230 MB/s at 16KB to ~520MB/s at 32KB (QD4). This has a tangable effect on OS and app performance. "
P.S. Question for GullLars. Noticed that Annihulus had writeback cache disabled on his setup. Is that best for intel raid setup on ICH10R? Dont know why he got in the 70k area with me in the high 50k area on vantage..

[ga1ve1an]

What about using a 4kb stripe size? Would that yield a better HDD score in Vantage?

Hey Musho didnt test the 4k Stripe... Depends on the queue depth that Vantage creates with the benchmark. GullLars also spoke of this in the link i posted above regarding anandtechs review. Here is his quote..

"Then you should be fine with a 97,5GB partition.
The reason smaller is better when it comes to stripe size on SSD RAIDs has to do with the nature of the storage medium combined with the mechanisms of RAID. I will explain in short here, and you can read up more for yourself you are more curious.

Intel SSDs can do 90-100% of their sequential bandwidth with 16-32KB blocks @ QD 1, and at higher queue depths they can reach it at 8KB blocks. Harddisks on the other hand reach their maximum bandwidth around 64-128KB sequential blocks, and do not benefit noticably from increasing the queue depth.

When you RAID-0, the files that are larger than the stripe size get split up in chucks equal in size to the stripe size and distributed amongs the units in the RAID. Say you have a 128KB file (or want to read a 128KB chunk of a larger file), this will get divided into 8 pieces when the stripe size is 16KB, and with 3 SSDs in the RAID this means 3 chunks for 2 of the SSDs, and 2 chukcs for the third. When you read this file, you will read 16KB blocks from all 3 SSDs at Queue Depth 2 and 3. If you check out ATTO, you will see 2x 16KB @ QD 3 + 1x 16KB @ QD 2 summarize to higher bandwidth than 1x 128KB @ QD 1.

The bandwidth when reading or writing files equal to or smaller the stripe size will not be affected by the RAID. The sequential bandwidth of blocks of 1MB or larger will also be the same since the SSDs will be able to deliver max bandwidth with any stripe size since data is striped over all in blocks large enough or enough blocks to reach max bandwidth for each SSD.

So to summarize, benefits and drawbacks of using a small stripe size:
+ Higher performance of files/blocks above the stripe size while still relatively small (<1MB)
- Additional computational overhead from managing more blocks in-flight, although this is negligable for RAID-0.
The added performance of small-medium files/blocks from a small stripe size can make a difference for OS/apps, and can be meassured in PCmark Vantage. "

[One_Hertz]

GullLars already confirmed that is not true based on how the Intel drives work. They Write Combine in blocks everything together instead of writing every little file each time which would wear faster(also known as write amplification). He commented on that on my original post at Anandtechs Raid Review here...

http://www.anandtech.com/Comments/sh...lComments=true

His quote

"There is no impact on write amplification from the striping. It can be explained pretty simply.
SSDs can only write a page at a time (though some can do a partial page write), and page size is 4KB on current SSDs. As long as the stripe size is above 4KB, you don't have writes that leaves space unusable.
With a 16KB stripe size, you will write 4 pages on each SSD alternating and in sequential LBAs, so it's like writing sequential 16KB blocks on all SSDs, and as the file size becomes larger than {stripe size}*{# of SSDs} you will start increasing the Queue Depth, but it's still sequential.

Since all newer SSDs use dynamic wear leveling with LBA->physical block abstraction, you won't run into problems with trying to overwrite valid data if there are free pages.

The positive side of using a small stripe is a good boost in files/blocks between the stripe size and about 1MB. You will see this very clearly in ATTO as the read and write speeds doubles (or more) when you pass the stripe size. F.ex. 2R0 x25-M with 16KB stripe jumps from ~230 MB/s at 16KB to ~520MB/s at 32KB (QD4). This has a tangable effect on OS and app performance. "
P.S. Question for GullLars. Noticed that Annihulus had writeback cache disabled on his setup. Is that best for intel raid setup on ICH10R? Dont know why he got in the 70k area with me in the high 50k area on vantage..

This is true only on a trimmed drive full of empty blocks that do not need to be erased... Also keep in mind you are increasing CPU usage substantially with smaller stripe.

[GullLars]

True CPU usage will be higher, but will it make a noteworthy difference? For 16KB vs 128KB, i think the added CPU overhead is made up for by acceleration of files in the 32-128KB range (in wich lots of the common OS IO requests fall).
I'm curious One Hertz, where do you have this info on write-combining and write-amplification suffering from 16KB stripe?
It's possible i've misunderstood the inner workings of the Intel controller, with clean block/page pools, and pools of dirty/semi-dirty waiting for cleaning.
As far as i understand, the controller will always write to the pool of free pages, and will not make much of a difference wich of the blocks the free pages belong to. The LBA that gets written will trigger the old LBA to be listed as dirty and to be cleaned at a later time, so if 1 LBA or 8 gets listed as dirty on the same time shouldn't matter.
Also, when it comes to 4KB write vs 32KB or 128KB, if you test ATTO or IOmeter, 4KB sequential @ QD 8 gets performance pretty simelar to 32KB @ QD 1 (wich is what a 32KB file will be written as respectivly on a 4KB and 32KB stripe). Each Intel SSD supports up to QD 32 nativly, so a 4KB stripe would limit you to 4KB sequential @ QD 32 bandwidth from each drive, wich is close to full sequential bandwidth. I don't have excact numbers, but it would be interresting to do a benching session on it. Unfortunately i don't have Intels myself, or i would do it.

I do not mind being proved wrong, as long as i learn something. If you have something i should read, pleese link it

[One_Hertz]

I am talking about when you run out of free pages, such as on a drive with no TRIM... For every small write it will be doing a full block erase (256k on intel drives? I don't remember). If there are free pages then you are correct of course.

Write combining makes this better by shoving lots of small writes into one block but it's not perfect.

[GullLars]

Either I or you have misunderstood how the Intel SSDs handle GC and spare area. I will not be as arrogant as to claim i am correct and you are not, cause i'm actually not certain, but i will present an argument for my view.

Let's take no TRIM as example. Even if you don't have TRIM, you will at least have 7% spare area, wich means there will at all times be 0-7% of the drives pages marked as "dirty" and the complement of the 0-7% will be clean (say f.ex. 5% dirty, 2% clean).
All clean pages will be gathered in blocks when blocks with dirty pages are erased, and the valid pages from the blocks that were erased will be written to some of the clean pages (this is the write amplification). After that, you will still have a bunch of clean pages wich you can write new data to, and all new data written will be to clean pages, erases will be triggered independently to make some "dirty" pages enter the "clean" pool (probably when a block passes a certain % dirty pages or the clean pool gets smaller than a certain % of total capacity). The size of the write will then only matter to transfer-time and array-time of the flash. Write-combining can be done when you have a number of outstanding pages to be written, and then stream many at once to be written physically sequentially in the same NAND chip. To get good array-time masking, 16-32KB blocks are good, but this can also be obtained with write-combining 4-8 outstanding pages and map the separated (or sequential) LBAs to the adjacent physical pages. The x25-V writes 4KB random at virtually the same speed as 1MB sequential, so a 4KB stripe size shouldn't matter at all for write speeds. Other SSDs like Indilinx Barefoot could suffer greatly since they do not write-combine this way, and that's reflected in the random write speeds. Sandforce and C300 seems to also do the same kind of write-combining with random write close to sequential write (for 4KB alligned writes).

[Computurd]

ty one hertz. even if the drive isnt trimmed, those blocks gotta be erased at one point or another, regardless. trim or not the block has to be erased.

[One_Hertz]

Gull is right as long as nothing big is written at one time to take up the available clean blocks. Im still stuck on pre-GC thinking because my X25-Es don't do anything like that.

edit: however I would like to note once again that write combining is far from perfect. It is only partially possible to combine the small writes into the same blocks. This is why intel states longevity as follows:

X25-M 160GB - 100% random host data 4k writes = 15TB
X25-M 160GB - 100% sequential writes = 370TB

[ga1ve1an]

Quick question for whoever may be able to answer this.. What type of GC does intels drive have? ANd how does it function? e.x. only on idle, low activity, or immediately.

Reason I ask is that with my Intel drives in Raid 0, I notice if I do excessive benchmarking it will give slower results.. but if i wait a few hours and run the benchmark again it seems to be at peak performance again.. So there is obviously GC working in Raid (which i guess is good since no Trim) But sometimes it seems to take longer than normal to get back to the high speeds. Again the slow down is not noticeable to me but is reflected in the benchmarks at least.

[GullLars]

The slowing down from benchmarking has 2 causes
#1. Depleting the clean page/block pool so you get GC speed (clean up, essentially a little write + erase) instead of write speed. The more random writes the drive has taken, the slower GC will restore performance, this is because it gets more difficult to get completly dirty blocks before cleaning, so you get more write overhead from moving valid LBAs to the clean blocks, and it takes longer time (more erases) to get out the dirty pages. Writing a large sequential file to all free space on the SSD and deleting it could help reduce this "fragmentation" quicker than idle GC.
Writing continualy sequentially to the SSD for some time should only result in a minor and temporary (seconds to minutes) decrease in write speeds to the sustainable erase-speed of the SSD, since blocks filled sequentially by the same file and the same LBAs being overwritten sequentially will result in entire dirty blocks with no or little write overhead.

The reason you likely won't feel the slight drop in performance is you're talking about fractions of a second in most IO-bound situations and a few seconds in most common sequential writes in everyday use. Read performance shouldn't take a noticable hit.

[Computurd]

well as one hertz notes, write combining is not perfect, and there is roughly a 25x faster wear rate when writing randomly than sequentially. so how do you think that would effect the wear by using a smaller stripe size? if the ssd is write combining, imperfectly of course, small stripes, upon deletion of that info there would have to be a erase cycle to fix that write. TRIM in effect is just cleaning (erasing) the block, correct? so even if it is trimmed it is still erasing a 256k block, correct?
so for example on a 4k stripe size it writes one 4k page. but in order to clear that write, erase it, wouldn't you still have to erase 256k?? maybe im not understanding correctly....

[GullLars]

You are ignoring LBA abstraction. When you want to overwrite a LBA, it doesn't get written to the same place, the controller simply marks the old data as dirty, and maps the LBA to a new physical page.
BTW, i see i only listed #1 in the last post, pardon me, i've been up for 24+ hours now, gaming for the last ~12 and have had 5 redbulls in that time :P
I see i said both, but didn't make the distinction.
#1 is free block/page depletion, #2 is the scattering of dirty LBAs from excessive random writes forcing GC to work harder.

Also, you need to make the distinction between sequential and random when talking about the impact on drive life. 4KB blocks written sequentially causes no more wear than 1MB blocks written sequentially (or at least not signifficantly more).
To take an example to illustrate. We postulate a 4R0 of x25-V with a 4KB stripe, and take a 1MB file as example.
The 1MB file will be split into 4KB chunks and evenly and sequentially distributed so each drive has every fourth LBA sequentially. This will give each drive 64 blocks of 4KB, essentially giving each drive 4KB Sequential write QD 64, wich is easily combined internally to f.ex. 8x 32KB blocks and sent to the flash, virtually maxing the write speed of each drive. When the file is deleted and overwritten sequentially, all LBAs will go dirty at the same time, and each SSD end up with a 256KB continous dirty area, wich gives little or no write overhead when GC comes to do it's job.
Given this logic, it should be more important for write amplification and GC on intel SSDs how pages go dirty (in larger clusters or scattered randomly all over) than how they are written. It's because it's harder to clean blocks whitout overhead than streaming arbitrary LBAs to pre-erased flash and re-mapping the table.

And BTW, intel cannot have 1.xx write amplification for pure 4KB random write if the 160GB only withstands 15TB of it compared to 370TB for sequential (regardless of block size). This would mean worst-case is ca 25x write amplification. This can be reduced dramatically with larger spare area (ref IDF whitepaper).

[Computurd]

The 1MB file will be split into 4KB chunks and evenly and sequentially distributed so each drive has every fourth LBA sequentially.

but what if the file is only 4k? when it goes to be deleted, isnt it still going to delete a 256k block? that is what i am trying to figure out.

*and as a side note, not sure about this either, but wouldnt a 4k stripe negate alot of the drives ability to write combine in the first place? say it is an eight member array and you write a 16k file. wouldn't that just increase the amount of disks that the information in that relatively small file is written too?
because then you have four drives with a 4k write (on each 4k page) on them, then you would have to delete a 256k block on each drive in order to clear that one write. so for writing 16k you have to delete a total of 1024k across the disks?

[GullLars]

No, when you delete a file, the SSD ignores it if TRIM isn't active. Nothing happens before the LBA (logic block, not physical) the file occupied is overwritten, at that point, the old position gets marked as "dirty" to be erased LATER by GC. GC selects blocks with a high % of dirty pages for cleaning, and copies the valid pages (if there are any) to a new fresh block.
Your fear of a 16KB file creating a 1024KB total write/erase an a 8R0 with 4KB stripe is irrational
One could say you have a bit of phobia/hysteria conserning write amplification

I thought i went through it in an earlier post, but intel SSDs keep a "clean" pool of blocks, and writes new data only to these, and maps the new LBA->physical location in a table. If it's indeed 256KB erase-blocks, then you can write a single 4KB block 64 times to each clean block, and only write one page each time whitout needing to erase anything. The LBA adress of such blocks is irrelevant, as the controller maps them on a table.

[Computurd]

OK now i get it...sorry sometimes i am dense...
thanks for your patience, and yes, i do have a phobia of wear on my array, these things are expensive