[Statistics] Reliability of various consumer SSD models

[minpayne]

Data source: Customer reviews at www.newegg.com

Methodology: Read through all reviews with only 3 eggs or less and identify deaths / problems through review description. (When a customer has met more than 1 dead SSD, the death count is only 1, and the review count is also only 1.)

Model	Review count	Review timespan	Total period (months)	Death count	Problem count (*)	Review failure rate	Annualized review failure rate (**)	Review problem rate
Intel X25-M G2 (still on sale)	232	Oct10-Oct11	12	6	2	2.59%	2.59%	0.86%
Intel X25-M G2 (deactivated)	276	Jul09-Jun11	23	6	3	2.17%	1.13%	1.09%
Intel 320 Series	453	Mar11-Jan12	10	29	13	6.4%	7.68%	2.87%
Intel 320 Series (excluding 8MB bricks with 0302 FW)	453	Mar11-Jan12	10	12	13	2.65%	3.18%	2.87%
Crucial M4 (excluding BSOD of 5184h bug)	1040	May11-Jan12	08	25	41	2.4%	3.6%	3.94%
Crucial M4 (64GB and 128GB)	889	May11-Jan12	08	15	32	1.69%	2.53%	3.6%
Crucial M4 (256GB and 512GB)	167	May11-Jan12	08	8	7	4.79%	7.19%	4.19%
Samsung 470	280	Aug10-Jan12	17	5	2	1.79%	1.26%	0.71%

(*) Problems include BSOD, freezing / stuttering issues etc.
(**) Assume the failures are evenly distributed over the power-on hours, which is what HDD vendors quote for MTBF.

I don't want to include the OCZ models, as the effort required to read through the negative reviews would be unacceptable!

Note that the statistics for G2 is incomplete - I am unable to obtain the links for all deactivated models (if you have a collection of them please let me know!)

The numbers of M4 stratified by page size don't add up exactly for the previous stats, probably because newegg activated / deactivated some products, or there were minor counting errors.

I'm inclined to believe that the 0302 FW of the 320 Series is prone to be bricked into 8MB, and the 8kb page sized models of the Crucial M4 (i.e. 256GB and 512GB) are less reliable.

My 2 cents.

[3lfk1ng]

Nice effort but the problem with this is that most consumers do not post reviews when they have a positive experience with a product. More people will post negative reviews when they have a bad experience with a product in order to provide warning. Therefore results are generally skewed towards the lower end.

This thread also fails to account for the following consumer variables. These are only ones that I could come up with and my experience with installing SSD's has taught me many things to look out for. Unfortunately, most of these variables cannot be accounted for through the internet. Please keep in mind that not everyone on Newegg is a computer enthusiast either:

• The consumers all have an SSD capable motherboard - even some older mobos will work with certain SSDs one out of every ~5 boot attempts.
• The consumers all have a controller capable of recognizing a SSD.
• The consumer's BIOS' support SSDs natively or they have all performed a BIOS update to add support.
• The consumers have all enabled support for AHCI and not IDE.
• The consumers didn't use Acronis to copy info from a mechanical HDD to the SDD (slower performance)
• The consumers all configured their SSDs correctly to get the speeds that the drive is capable of.
• The consumers could possibly be complaining of speed issues because they are on SATA II capable motherboards or using the SATA II ports instead of SATA III.
• The consumers *sigh* you get the point.

In my opinion this thread is dead because of these unaccountable variables. Valiant attempt though, at least we know which SSD controllers are more prone to issues (I'm looking at you Intel, Sandforce).

[minpayne]

Nice effort but the problem with this is that most consumers do not post reviews when they have a positive experience with a product. More people will post negative reviews when they have a bad experience with a product in order to provide warning. Therefore results are generally skewed towards the lower end.

I wouldn't worry about the results skewed towards the lower end as we can still distinguish between products.

• The consumers all have an SSD capable motherboard - even some older mobos will work with certain SSDs one out of every ~5 boot attempts.
• The consumers all have a controller capable of recognizing a SSD.
• The consumer's BIOS' support SSDs natively or they have all performed a BIOS update to add support.
• The consumers have all enabled support for AHCI and not IDE.
• The consumers didn't use Acronis to copy info from a mechanical HDD to the SDD (slower performance)
• The consumers all configured their SSDs correctly to get the speeds that the drive is capable of.

These are counted as to evaluate the compatibility of SSDs, which is reflected in the "Problem rate" column. The end-user experience is a very important factor imo, while most SSD vendors now focus on how to "cheat" scores to attract noobs. If you are looking for a "trouble-free" SSD, you know which ones to take 4k-alignment may be an issue, but I assume this is evenly distributed among different SSD models still.

• The consumers could possibly be complaining of speed issues because they are on SATA II capable motherboards or using the SATA II ports instead of SATA III.
• The consumers *sigh* you get the point.

This is why I said I *read* through the reviews. I ignored all these kind of comments.

[Ao1]

I suspect that feedback might also be influenced by the timing of a problem in relation to the purchase date, i.e. someone is more likely to report a problem on NewEgg within the first weeks of purchase and perhaps less likely to report a problem after 12 months.

As there is more published info on HDD failures it might be worth checking NewEgg to see if HDD reviews tie up with published failure rates. That would help give a perspective on what is being reported for SSD reviews.

[johnw]

I wouldn't worry about the results skewed towards the lower end as we can still distinguish between products.

Whenever I have mentioned something like this in the past, I have found it odd that so many people are unable to grasp the simple concept that while you cannot determine an ABSOLUTE failure rate from this sort of data, it is still quite useful to compare RELATIVE failure rates of different products with this data. In fact, it is probably the best method available for comparing relative failure rates of products when making buying decisions, since the retail return rate data is so out of date by the time it is available.

But I think your presentation in the first post of this thread has unneccesarily confused the matter. I would definitely not label the columns "failure rate" or "annualized failure rate", since that phrasing suggests that you are determining the percentage of products sold that fail. But you are only determining the percentage of products REVIEWED that fail. So it should really have another name. My preference would be to call it something like "failure index" or "review-reported failure rate".

[Computurd]

this is a valiant effort, and i applaud your efforts.

however, this data is anecdotal, at best. basically i do not think that any assumptions or valuations of 'worthiness' can be made from this data.
this is basically not much better than guessing imho. you have no comparative basis for number of units, number of *possible* respondents, etc. this is just vague data.
the user is the source of the vast majority of computer related issues, hence the adoption of the PEBKAC saying amongst professionals.
especially in an area such as this, where the majority of users have no idea what in the world they are doing.
we arent gauging the devices here, we are gauging the inability of people to use their hardware, and sprinkling in 1 actual failure for every 5 reported 'incidents'.
i see this stuff in forums so constantly, it drags me down. people complaining, and you explain explain explain, then they fix it.
bad thing is, most dont even post in forums, they just RMA. if they took the time to learn it would be easy to fix.
so some sites (the mainstream overclocking sites especially) just get clogged with hundreds and thousands of these "my ssd doesnt work" threads and in my experience, 1 in 20 is an actual problem that doesnt lie between the users ears.
i guess im railing against the machine here LOL
but its tiring, and i avoid those threads like the plague!

just my .02c

[minpayne]

I'm still not convinced that the distribution is biased. Unless someone can prove that more noobs have chosen the "less reliable" models, or the noise of uncertainty is greater than a certain scale.

[johnw]

I'm still not convinced that the distribution is biased. Unless someone can prove that more noobs have chosen the "less reliable" models, or the noise of uncertainty is greater than a certain scale.

I've not seen any evidence that the data is biased in such a way that it makes RELATIVE comparisons of SSD reliability suspect.

Most people that complain about that cannot grasp basic probability and statistics, and they just make ignorant claims like "users are idiots so the review data is useless". But such complainers completely miss the fact that the expertise of the users only matters for relative comparisons if there are a significantly larger percentage of clueless reviewers for one product than for another product.

Besides, even if there were a larger percentage of reviews from clueless users for brand X, I'm not sure that is a good selling point for brand X. "Buy brand X, thousands of clueless users cannot be wrong! ....Er, perhaps clueless people could be wrong, but at least you'll be in good company! ...Uh, maybe not GOOD company. I suppose you will be in bad company, but at least you will not be alone ....Ahem, so buy brand X! If you are clueless enough, you might like it! ...or maybe not."

[Computurd]

"Buy brand X, thousands of clueless users cannot be wrong!

i love it! i know some marketing people who would actually ponder doing that for a moment. that did give me lols for real

expertise of the users only matters for relative comparisons

my belief is that there are just too many intangibles. regardless of anything, i personally feel that you cannot make sound judgments of any sort based upon unreliable data.

if unreliable data that is vague and ambiguous, at best, is the cornerstone of an application of probability and statistics....well....you can imagine where i am going with this

[minpayne]

More people will post negative reviews when they have a bad experience with a product in order to provide warning. Therefore results are generally skewed towards the lower end.

A simple proof of why it still works to compare between products:

Assume that the total amount sold is T, the true failure rate is f, Pr(customer gives positive feedback | no problem) = a, Pr(customer gives negative feedback | problems) = b, according to your claim we have a < b.

Positive feedback count = T * (1 - f) * a
Negative feedback count = T * f * b

My estimated failure rate F = Negative feedback count / (Positive feedback count + Negative feedback count) = (T * f * b) / (T * (1 - f) * a + T * f * b) = 1 / [ a/(b*f) + (1-a/b)]

Note that F is a function of f, which is monotonically increasing on the interval (0, 1). Hence worse products would get worse estimated failure rate, which is why it's still valid to compare between products.

[Ao1]

I agree it gives a perspective. It’s always better to triangulate as many sources of information as possible to enhance the validity of the perspective and a good way to do that is to compare reviews with a product that has established failure rates. If the reviews coincide with published failure rates it helps to validate reviews on a product with unknown failure rates. A good starting point is the Spinpoint F3, which has 1951 reviews.

http://www.newegg.com/Product/Produc...Spinpoint%20F3

Edit:

OCZ Technology VTX3-25SAT3-120G – 500 Reviews

26% below satisfactory (2 eggs or below)
69% above satisfactory (4 eggs or above)

SAMSUNG Spinpoint F3 HD103SJ 1TB 7200 RPM – 1,951 Reviews

14% below satisfactory (2 eggs or below)
81% above satisfactory (4 eggs or above)

Intel 320 Series SSDSA2CW120G3B5 2.5" 120GB – 27 reviews

18% below satisfactory (2 eggs or below)
74% above satisfactory (4 eggs or more above)

[Computurd]

Assume that the total amount sold is T, the true failure rate is f, Pr(customer gives positive feedback | no problem) = a, Pr(customer gives negative feedback | problems) = b, according to your claim we have a < b.

Positive feedback count = T * (1 - f) * a
Negative feedback count = T * f * b

My estimated failure rate F = Negative feedback count / (Positive feedback count + Negative feedback count) = (T * f * b) / (T * (1 - f) * a + T * f * b) = 1 / [ a/(b*f) + (1-a/b)]

no amount of math is going to account for the fact that the vast majority of users who are happy NEVER post back. i never have. not once, and i have bought literally hundreds, if not in the thousands, of products in the last few years.

unless you know what percentage off users aren't replying, instead of literally guessing, the whole premise of the exercise is pointless.

its like building a house on sand. no matter what you build, how strong it is, the foundation still consists of sand.

and no matter what predictions are made, they are baseless unless we were to know how many units are sold.

imo posting 'statistics' such as these is akin to fearmongering. these types of skewed numbers really arent going to do much for the furtherance of SSDs as a whole when some guy stumbles across them who takes them as absolute.


it yields ridiculously skewed results, even when compared to other numbers that are out there, for example the retailers who provide good data, and the numbers from the manufacturers themselves. these results are worlds apart.

[minpayne]

no amount of math is going to account for the fact that the vast majority of users who are happy NEVER post back. i never have. not once, and i have bought literally hundreds, if not in the thousands, of products in the last few years.

unless you know what percentage off users aren't replying, instead of literally guessing, the whole premise of the exercise is pointless.

This is why we assume variables a and b, where a<b, 0<a<1 and 0<b<1. What's wrong with analytically deriving formulae, unless you hate math?

its like building a house on sand. no matter what you build, how strong it is, the foundation still consists of sand.

Then tell me which part of my proof is invalid or not solid? I'm listening

and no matter what predictions are made, they are baseless unless we were to know how many units are sold.

As shown in my proof, total number of drives sold T is cancelled, which is not useful here.

imo posting 'statistics' such as these is akin to fearmongering. these types of skewed numbers really arent going to do much for the furtherance of SSDs as a whole when some guy stumbles across them who takes them as absolute.

it yields ridiculously skewed results, even when compared to other numbers that are out there, for example the retailers who provide good data, and the numbers from the manufacturers themselves. these results are worlds apart.

Proof is already given. Make whatever you like There's one thing you should know about: in Statistics, you never get 100% reliable data, but the most important thing is how you work through it and interpret.

[johnw]

minpayne:

Your simple computation is clear and correct, and your argument is logical and almost certainly true -- the newegg review data can provide useful information when properly analyzed. I only wish you would correct the misleading column headings in your first post.

It would also be helpful to post your criteria for evaluating the reviews and sorting them into "failure" and "problem" categories. Along with some examples of actual reviews and the categories you assigned. Without that information, it is not possible for others to add to your work since they may unwittingly use different criteria in assigning reviews to the "failure" or "problem" categories.

[[XC] Oj101]

There was a thread here with failure rates based on returns vs units shipped, and Intel had the lowest return rate. They're also known for building incredibly reliable SSDs, yet your list shows them to be a brand to avoid. Is that proof enough?

[johnw]

There was a thread here with failure rates based on returns vs units shipped, and Intel had the lowest return rate.

If you are referring to BeHardware's component return rates survey,

http://www.behardware.com/articles/8...s-rates-5.html

then you really need to pay more attention to detail. From that article:

"The returns rates given concern the products sold between October 1st 2010 and April 1st 2011 for returns made before October 2011, namely after between 6 months and a year of use."

As you should know, the Intel 320 series was not even being sold until late March 2011, so it is hardly covered by the BeHardware survey at all. Furthermore, the Intel 320 series had (or has) a nasty bug, dubbed the "8MB bug", that causes complete data loss. Intel issued a revised firmware to address that issue in August, 2011, but there have been continuing reports of 8MB issues even with the revised firmware. The continuing reports may not be the same issue, since it appears there may be more than one error that can cause the SSD to revert to 8MB capacity, but whatever the cause, some people are still having problems.

So, it is plausible that the Intel 320 SSDs could have higher failure rates than some other products, such as the Intel G2, Crucial, or Samsung SSDs. Also, if you look at the Intel 320 SSDs "excluding 8MB bricks with 0302 FW", then the failure index is lower than the Crucial m4. And I suspect that if minpayne (or anyone) applied the same methodology to the Sandforce 2281 SSDs, they would find an even higher failure index than the Intel 320 series.

Finally, I fail to see the logic of the argument that puts greater trust in the product return rate survey than in the newegg reviews. Why would anyone assume that there is less chance of a difference in average user expertise between products in the decision to return a product or not than in the decision to post a bad review or not? I'd actually put more trust in the newegg reviews, since you can read the reviews and judge for yourself whether there was an actual failure. But with the return rate survey, all you know is that the product was returned.

[[XC] Oj101]

The 8 MB bug affected what, 1 % of drives sold? Out of dozens I've sold, not one has come back. The 8 MB bug is made out to be something that affects 50 % or more drives, this is not the case. It got a lot of publicity due to Intel having a relatively good reputation and having a hiccup.

As for your last point, that has already been covered in this very thread.

[minpayne]

@Ao1: That's a good suggestion, however that amount of effort required to read through 361 less-than-4-egg comments is not what I can devote now, not until I can automate the process!

[minpayne]

There was a thread here with failure rates based on returns vs units shipped, and Intel had the lowest return rate. They're also known for building incredibly reliable SSDs, yet your list shows them to be a brand to avoid. Is that proof enough?

I bet you are not referring to this survey are you? Their sample size is even smaller than mine.

[minpayne]

The 8 MB bug affected what, 1 % of drives sold? Out of dozens I've sold, not one has come back. The 8 MB bug is made out to be something that affects 50 % or more drives, this is not the case. It got a lot of publicity due to Intel having a relatively good reputation and having a hiccup.

As for your last point, that has already been covered in this very thread.

1% of drives sold? Where is your data from?

From the newegg data I'm inclined to believe that the 0302 FW of the 320 Series is buggy. Also, the reliability of the Crucial M4 256GB/512GB models (8k page sized) is questionable.

Even after Intel has fixed the BAD_CTX 13X bug with the 0362 FW, people still have a chance to brick their 320 Series into 8MB: http://communities.intel.com/thread/24339?tstart=0

Actually 8MB seems to be a general failure mode for all Intel SSDs, e.g. 710 Series getting 8MB: http://docs.oracle.com/cd/E23247_01/pdf/E23248.pdf

Here is an important point we'd like to know: does the 320 Series suffer from elevated probability of bricking into 8MB (not counting the BAD_CTX 13X bug in 0302 FW), compared against the X25-M G2, possibly due to the lower endurance / reliability of the 25nm nand flash?

[johnw]

The 8 MB bug affected what, 1 % of drives sold?

You make up a number out of thin air, and assume that is more trustworthy than the numbers that minpayne has computed from careful analysis of newegg reviews? Wow, with irrationality like that, why should anyone even bother to read a single word that you write?

[minpayne]

@johnw:

I can't reply to any of your posts here cuz they are all well said and there's no bit for me to add!

[Manicdan]

my simple 2 cents

to determine sales go by number of 4/5 eggs + 50% of 3 eggs + 20% of 1/2 eggs. this is because people would be more inclined to post with bad news rather than good.
then using 1/2 eggs as a quick % of issues/failures. by now we should have a clear indication of what % of people who post bad news also had a legitimate issue.

yes this is overly simplified but i think can quickly give a relative comparison between a large number of products.

[[XC] Oj101]

You make up a number out of thin air, and assume that is more trustworthy than the numbers that minpayne has computed from careful analysis of newegg reviews? Wow, with irrationality like that, why should anyone even bother to read a single word that you write?

Simple Googling and looking for the total number of people plagued with the issue in comparison to the number of drives sold. Going by reviews is hardly the way to do it and my way isn't much better either. Best would be to look at return rates due to failure in relation the number of units shipped - something which you aren't going to get from NewEgg or any forums.

[minpayne]

Simple Googling and looking for the total number of people plagued with the issue in comparison to the number of drives sold. Going by reviews is hardly the way to do it and my way isn't much better either. Best would be to look at return rates due to failure in relation the number of units shipped - something which you aren't going to get from NewEgg or any forums.

Googling with what? What's the total number of drives sold, and the number of people plagued with the issue then?