Anandtech's secret method for computing Write Amplification of SSDs

[johnw]

According to a writer for Anandtech.com, Kristian Vättö, they have a secret method for computing write amplification (WA) for any SSD:

The method we use was disclosed by a big SSD manufacturer a few years ago. It does not rely on SMART or power consumption, and it can be run on any drive.

....

Also, our method is just one way of estimating worst case write amplification.

Any guesses or speculation on what they are doing?

[thegoatman]

What is Write Amplification?

[johnw]

http://lmgtfy.com/?q=write+amplification

[Anvil]

I don't see how one can get anything at all w/o reading/checking some sort of status off of the drive. (before and after some predetermined I/O)

Host writes (measured from the app) is not necessarily the same as Host writes on the drive as the "system" can do write combining/coalescing and this just adds to the task of finding the correct WA.

"Worst case" might be easier to get to, but still, I can't think of any way of knowing w/o checking the drive.

[johnw]

"Worst case" might be easier to get to, but still, I can't think of any way of knowing w/o checking the drive.

I can see how you might be able to get into the ballpark using a power consumption method (although I don't know how accurate it would be), but Vättö claims they aren't using power consumption (or SMART attributes).

I wonder if it could be done by secure erase, then doing full-span random writes and observing the write speed as it decays. Maybe you can deduce when it ran out of pre-erased flash that way. But this method could be problematic if the SSD has aggressive GC, since the write speed decay curve may be "noisy" and the GC could be recovering some flash even before all of the flash gets written the first time.


Here's an example of WA as measured using anandtech.com's secret method:

[Computurd]

my question is this: why is it secret? is it because it will not pass muster? seriously, there is no need for any website to have 'secret' testing methods. results should be reproducible in order to be believable.
Im thinking that they have figured out that their method may be flawed, and do not want to take the chance of a blow to credibility. no reason whatsoever for smoke and mirrors.
This creates a credibility problem.

EDIT: many SSDs have a percentage of fill that triggers GC, such as 90%, etc. Once filled to that level the GC kicks in, which would invalidate that method on many SSDs.

[Zaxx]

Gotta agree with Computurd on this....all testing methodology needs to be fully disclosed and reproducible. That's the basic foundation for any review imo.

[johnw]

I wonder if it could be done by secure erase, then doing full-span random writes and observing the write speed as it decays. Maybe you can deduce when it ran out of pre-erased flash that way. But this method could be problematic if the SSD has aggressive GC, since the write speed decay curve may be "noisy" and the GC could be recovering some flash even before all of the flash gets written the first time.

It was suggested to me that anandtech's method may just be:

WA = FOB write speed / worst case write speed

which I am assuming the write speed would be measured for 4K random write (probably high QD). Obviously this formula will not be entirely accurate depending on the details of how the firmware handles GC.

Whether anandtech.com is using that method, or something like the method I quoted above, it basically depends on measuring the write speed decay over time, starting with a fresh (secure-erased) SSD.

Personally, I'd much rather just see the graph of write speed over time shown in the review, and let me draw my own conclusions, rather than having anandtech.com look at the graph themselves and draw some possibly erroneous conclusions about write amplification.

[Highendtoys]

my question is this: why is it secret? is it because it will not pass muster? seriously, there is no need for any website to have 'secret' testing methods. results should be reproducible in order to be believable.
Im thinking that they have figured out that their method may be flawed, and do not want to take the chance of a blow to credibility. no reason whatsoever for smoke and mirrors.
This creates a credibility problem.

EDIT: many SSDs have a percentage of fill that triggers GC, such as 90%, etc. Once filled to that level the GC kicks in, which would invalidate that method on many SSDs.

I have already rubbed off on you!!!

This stuff isn't rocket science with a bunch of proprietary secrets. Just like all of the secrecy around the "real world" test suite that Anand and a couple of others are doing, if others can't reproduce the results when what good is it to the readers? When I started fill testing I went right into full disclosure mode and wrote an article telling everyone exactly how I did it with links to every single file I use. I even jumped through hoops to build the test with files that were easily accessible by everyone. I know of at least three manufactures/resellers that now fill test their products using the same format that I use.

[Computurd]

possibly erroneous conclusions about write amplification

Absolutely erroneous imo. There is no way to know what other background processes are taking place that are affecting the write speed. GC, wear leveling, ecc. There are also other algorithms for read disturb, static data rotation, etc. there is just so much going on that there is no way, if they are using this method, that it is accurate.

[johnw]

Vättö states in a comment that the reason their WA chart does not show any Sandforce SSDs is because the worst-case "WA" comes out to be about 2 for Sandforce SSDs.

Yet more evidence that the method they are using is not accurate. It looks to me like they are measuring write speed with HD Tune, and probably just measuring fresh write speed, torturing the SSD with a lot of 4K random writes, then running HD Tune again to get "worst case" write speed.

No wonder Anand is afraid to reveal the method. It is a joke.

[johnw]

One thing I'd like to see in all SSD reviews would be a graph of write speed vs. time (or vs. GB written) for three cases, using random data being written to the SSD, and a secure erase before each case:

1) Sequential write of the full capacity of the SSD, every LBA in order, twice (second time to see how the fully-written SSD behaves)

2) full-span 4K random write, QD=1, just keep writing until the write speed stops decreasing and reaches steady-state

3) same as 2, but with QD=32

Of course this wouldn't give an accurate measurement of WA, but it would still provide a lot of useful information about how the SSDs behave under very heavy, worst-case conditions. It would also probably expose a lot of bugs and design flaws in the SSD firmware (for the SSDs that have such problems).

[Highendtoys]

One thing I'd like to see in all SSD reviews would be a graph of write speed vs. time (or vs. GB written) for three cases, using random data being written to the SSD, and a secure erase before each case:

1) Sequential write of the full capacity of the SSD, every LBA in order, twice (second time to see how the fully-written SSD behaves)

2) full-span 4K random write, QD=1, just keep writing until the write speed stops decreasing and reaches steady-state

3) same as 2, but with QD=32

Of course this wouldn't give an accurate measurement of WA, but it would still provide a lot of useful information about how the SSDs behave under very heavy, worst-case conditions. It would also probably expose a lot of bugs and design flaws in the SSD firmware (for the SSDs that have such problems).

I think I can do that in Passmark 7. Let me finish the tests that I'm running on a drive now and give it a shot.