Results 1 to 10 of 10

Thread: 6TB drives are good for your data: discuss!

  1. #1
    Xtreme Member IanB's Avatar
    Join Date
    Jan 2007
    Location
    Dorset, UK
    Posts
    439

    6TB drives are good for your data: discuss!

    There's been a couple of reviews on Anandtech recently about new 6TB drives in consumer-class and enterprise variants: eg. http://anandtech.com/show/8263/6-tb-...ate-ec-hgst-he That review in particular disturbed me in that a consumer-class drive (the WD Red) was discussed alongside two enterprise-class drives as suitable for home NAS use with no mention of the high chance of read errors due to the sheer size of the beasts.

    I want to copy here the maths I laid out in a comment on that article, and maybe provoke a discussion. Since this is an Xtreme forum, my guess is that many people may be thinking that bigger/newer is better, salivating at the idea of 6TB in a single package and building huge arrays, and thinking that a RAID6 will keep their data (relatively) safe. Or even thinking they are OK for use in a NAS in a mirror set, for instance. I'm not discussing the need for backups here, just the issue of keeping the array running and being able to rebuild in the case of drive failure. It's possible that running ZFS will mitigate this issue, but these drives will no doubt be used by misinformed users in normal RAID or mirror sets and that's where the danger lies.

    The most important number for this discussion is the URE figure for a drive, or unrecoverable bit read error rate. For normal consumer-class drives this is quoted at 1 in 10^14 bits, and for enterprise-class drives it is 1 in 10^15 bits. This is partly why those drives (as well as being designed/marketed specifically for 24/7 array usage) are more expensive. As a first reference, read these great articles which lay out the core of the problem: http://www.zdnet.com/blog/storage/wh...ng-in-2009/162 and http://www.zdnet.com/blog/storage/wh...ng-in-2019/805

    So here's the maths which (to me) shows that 6TB drives are completely crazy in consumer-class ranges.

    6TB is approximately 0.5 x 10^14 bits. That means if you read the entire disk (as you have to do to rebuild a parity or mirrored array from the data held on all the remaining array disks) then there's a 50% chance of a disk read error for a consumer-class disk with 1 in 10^14 unrecoverable read error rate. Conversely, that means there's a 50% chance that there WON'T be a read error.

    Let's say you have a nice 24TB RAID6 array with 6 of these new 6TB WD Red drives - four for data, two parity. RAID6, so good redundancy right? Must be safe! One of your disks dies. You still have a parity (or two, if it was a data disk that died) spare, so surely you're fine? Unfortunately, the chance of rebuilding the array without ANY of the disks suffering an unrecoverable read error is: 50% (for the first disk) x 50% (for the second) x 50% (for the third) x 50% (for the fourth) x 50% (for the fifth). Yes, that's about 3% chance of rebuilding safely. Most RAID controllers will barf and stop the rebuild on the first error from a disk and declare it failed for the array. Would you go to Vegas to play those odds of success?

    If those 6TB disks had been enterprise-class drives (say WD RE, although there's no RE 6TB drive yet, or the HGST and Seagates reviewed by Anandtech) they would have a 1 in 10^15 unrecoverable bit read error rate, an order of magnitude better. How does the maths look now? Each disk now has a 5% chance of erroring during the array rebuild, or a 95% chance of not. So the rebuild success probability is 95% x 95% x 95% x 95% x 95% - that's about 77.4% FOR THE SAME SIZE OF DISKS. Obviously you've invested more heavily in better disks to achieve that, but the end result is that you have a better-than-even chance of completing an array rebuild if you lose a drive!

    Note that this success/failure probability is NOT PROPORTIONAL to the size of the disk and the URE rate - it is a POWER function that squares, then cubes, etc. given the number of disks remaining in the array. That means that using smaller disks than these 6TB monsters is significant to the health of the array, and so is using disks with much better URE figures than consumer-class drives, to an enormous extent as shown by the probability figures above.

    For instance, suppose you'd used an eight-disk RAID6 of 4TB Red drives to get the same 24TB array in the first example. Your non-error probability per disk full read is now roughly 65% (better since there are fewer bits being read), so the probability of no read errors over a 7-disk rebuild after a single failed drive is 65% x 65% x 65% x 65% x 65% x 65% x 65% or roughly 5%. Better than 3%, but not by much. The same calculation using 2TB disks for a 13-disk rebuild of a 14-disk 24TB array gives 8% - not much better, unfortunately. However, all other things being equal, using far smaller disks (but more of them) to build the same size of array IS marginally safer for your data.

    Before anyone rushes to say none of this is significant compared to the chance of a drive mechanically failing in other ways during the rebuild, sure, that's an ADDITIONAL risk of array failure to add to the pretty shocking probabilities above. The bottom line for me is that anyone thinking of using huge drives and just adding extra redundancy drives to parity arrays, to make up for the fact they are using consumer-class drives, is ignoring the fact that their brag-worthy huge drives will likely fail and kill the array as soon as it is stressed during a full rebuild after any drive failures. Consumer-class drives are intrinsically UNSAFE for your data at these bloated multi-terabyte sizes, however much you think you're saving by buying the biggest available, since the build quality has not increased in step with the technology cramming the bits into smaller spaces.

    I'm extremely disappointed that the Anandtech review completely ignored this fact and didn't recommend that the 6TB Red drives were a seriously risky proposition for the Home/NAS usage they were being reviewed for. Especially as the review even reported an array rebuild failure using 6TB WD Reds that could well have been such a bit read error, merely dismissing it as a possible "compatability issue" with the particular NAS. EDIT - Ganesh, the Anandtech reviewer, has since responded personally to me showing the error was not a URE on the rebuild, but a odd write error apparently happening afterwards as the rebuilt array was being tested.

    Without a serious improvement in the build quality/URE figures for these huge drives, I feel the conclusion of the article should have been a very clear warning that these consumer-class 6TB drives should be avoided at all costs. Despite the obvious technological and engineering achievement in squeezing 6TB into standard-size HD packages, my gut instinct is summed up by the Jurassic Park quote: "your scientists were so preoccupied with whether or not they could that they didn't stop to think if they should."
    Last edited by IanB; 07-23-2014 at 01:19 PM. Reason: Information from Anandtech reviewer
    Quote Originally Posted by Particle View Post
    Software patents are mostly fail wrapped in fail sprinkled with fail and sautéed in a light fail sauce.

  2. #2
    Xtreme Member
    Join Date
    Mar 2011
    Posts
    150
    It is a concern with these disks. I currently have 9 x 4TB in RAID6 and various 3TB, 2TB and 1TB disks in RAID6 and RAID5.

    I think for me personally I'm done with home storage and over the next few years I'm going to transition to maintaining a much smaller home storage pool for personally created content (Video, Pictures, Documents, Work stuff etc) and then turn to streaming services such as Hulu, Netflix, Spotify, Beats Music for content.

    Purchasing and storing media at home when you have a lot is becoming quickly precarious as content sizes increase due to HD and now 4K on the horizon and the low life span and infant mortality rate of hard disks. I don't want to pay double the price of a consumer disk just to get acceptable durability and I wont.

  3. #3
    Xtreme Member IanB's Avatar
    Join Date
    Jan 2007
    Location
    Dorset, UK
    Posts
    439
    Personally, for many years, ever since I discovered the difference between consumer and RE/Enterprise drives, I've never bought a consumer-class drive or installed one for my friends or at work. I'm willing to pay and recommend that premium for a good reason.

    Now that SSDs (that have virtually no asssociated mechanical failure risks and draw significantly lower power) are reaching 1TB at affordable prices and 2TB is likely to arrive fairly soon, I suspect mechanical drives are going to have a very short market lifespan. After all, what is the effective URE rate for SSDs? I'm simply not planning any RAIDs with mechanical disks any more.
    Quote Originally Posted by Particle View Post
    Software patents are mostly fail wrapped in fail sprinkled with fail and sautéed in a light fail sauce.

  4. #4
    Crunching For The Points! NKrader's Avatar
    Join Date
    Dec 2005
    Location
    Vancouver WA, USA
    Posts
    2,886
    Quote Originally Posted by IanB View Post
    Personally, for many years, ever since I discovered the difference between consumer and RE/Enterprise drives, I've never bought a consumer-class drive or installed one for my friends or at work. I'm willing to pay and recommend that premium for a good reason.

    Now that SSDs (that have virtually no asssociated mechanical failure risks and draw significantly lower power) are reaching 1TB at affordable prices and 2TB is likely to arrive fairly soon, I suspect mechanical drives are going to have a very short market lifespan. After all, what is the effective URE rate for SSDs? I'm simply not planning any RAIDs with mechanical disks any more.
    Because ssd don't have a maximum amount of writes or anything.

    Also, 1tb ssd affordable? For who? Bill gates or Oprah?
    I'm having a hard time affording a 900$ worth of mechanical drives to think that I would at least need to quadruple that number to get any real amount of data wouldn't even be worth my time to dream about, apart from rich people and enterprise mech drives will be around a long time as most people would rather run that chance of failure than either a. Not get a storage array in the first place or b. Spend 5x as much (if not more)

    This failure rate thing has been around for a while, and is well known, this is why anyone who had a raid array rarely runs over 2tb drives, these 6tb drives are for random gaming rigs running one storage drive etc.

    MainCrunch
    Case : CaseLabs Mercury S5 | Heatsink : Corsair H80i | Motherboard : | CPU : | Ram : |Storage : Samsung 840 Pro 256GB - 2x WD Black 2TB Raid1 | PSU : SeaSonic Platinum 1000w | GPU : Sapphire R9 290x | Monitor : ASUS VG248QE

    Crunchy Fileserver
    Supermicro CSE-732i-500B | Supermicro H8DMi-2 | 2x AMD Opteron - 2419ee | 2x Cooler Master Hyper T2 | 6GB DDR2 REG ECC | Samsung Spinpoint 500GB - JBOD Pile of drives
    →People Crunching4Me←
    2x Opteron 2419ee | Intel Core2Quad Q8300 | Intel i7 920 | Intel Core2Quad Q6600

  5. #5
    Xtreme Member IanB's Avatar
    Join Date
    Jan 2007
    Location
    Dorset, UK
    Posts
    439
    Quote Originally Posted by NKrader View Post
    Because ssd don't have a maximum amount of writes or anything.
    Most end users (not companies) here aren't building arrays for high-throughput database work, but Write-Once Read Many (WORM) media storage arrays. Even with significant daily throughput the (non)issue of maximum write limits being reached within the likely lifespan of most SSD devices has been debunked over and over. You're just way out of touch there, unless you're doing something extreme with your arrays, in which case you wouldn't be using consumer-class drives in any case, which is what my original post was all about. So totally not relevant.

    Also, 1tb ssd affordable? For who? Bill gates or Oprah?
    Right now, 1TB SSDs are available at a price point where 200GB drives were just a few years ago. Samsung, one of the biggest makers, has 3D NAND tech in the market now that will drastically reduce the cost of huge NAND arrays as it filters into more products, as it doesn't rely on expensive sub-20nm lithography to achieve density. Comparing the lowest-cost 1TB RE-class drive I can see on a good UK retailer, the price differential is only 4 times now, not "5x or more". That's not at all bad for relatively new and SIGNIFICANTLY faster and more reliable tech, and things will only get better, quickly now...

    This failure rate thing has been around for a while, and is well known, this is why anyone who had a raid array rarely runs over 2tb drives
    If it's that obvious and well-known, then why did this review not mention it? My guess is that there are a great many new users, browsing this forum and others and sites like Anandtech, perhaps looking at big drives to build home NASs and media servers which are all the rage now, who simply aren't aware of this. (And maybe the reviewers aren't either, or are less focussed on it now). That's why I flagged it, because what I read was a prominent review of new tech that lauded the achievement of stuffing more data in an unreliable container without explaining the serious drawbacks of that when the fundamental build quality is unchanged. That's even more reprehensible when the nature of tech marketing is to exploit younger or less informed users looking for e-peen and bragging rights, which these massive drives apparently supply in spades. Isn't that what forums like this are designed to counter, with useful information to help inform choice?

    All I want to point out (and invite discussion about) is that despite the marketing and reviews which might suggest these huge consumer-class drives are useful and acceptable, they are simply NOT a solution to a problem of increased need for storage if they merely become bigger and more expensive points of more likely failure as a consequence of their stagnant, high URE rate. Although they appear "efficient" they are a completely false economy and WILL cost more in the long run due to replacement or loss of data compared to paying the premium for (preferably smaller) enterprise quality disks in the first place.
    Quote Originally Posted by Particle View Post
    Software patents are mostly fail wrapped in fail sprinkled with fail and sautéed in a light fail sauce.

  6. #6
    Xtreme X.I.P. knopflerbruce's Avatar
    Join Date
    Oct 2006
    Posts
    1,254
    My first thought when reading this is that the URE specs don't necessarily have to be realistic. From what I've seen they haven't changed for ages. That just sounds strange. It's an interesting topic, but it would be better if we had some hard facts regarding URE's rather than product specs.

    FWIW, the HGST SAS drives are another order of magnitude better, 10^16 This gives a 2.5% chance of URE rebuild FAILURE if these were available as 6TB drives, and one out of six failed in a RAID6 array.
    --->TeamPURE<---

  7. #7
    Xtreme Member IanB's Avatar
    Join Date
    Jan 2007
    Location
    Dorset, UK
    Posts
    439
    Quote Originally Posted by knopflerbruce View Post
    FWIW, the HGST SAS drives are another order of magnitude better, 10^16 This gives a 2.5% chance of URE rebuild FAILURE if these were available as 6TB drives, and one out of six failed in a RAID6 array.
    I wish that were true... I think you misread the same spec sheet I saw, which listed it (cleverly) as 10 in 10^16. The spec is indeed 1 in 10^15 as shown in their datasheet: http://www.hgst.com/tech/techlib.nsf...e/USHe6_ds.pdf

    That helium-filling tech looks great, though. I hope they introduce it on smaller drives too.
    Quote Originally Posted by Particle View Post
    Software patents are mostly fail wrapped in fail sprinkled with fail and sautéed in a light fail sauce.

  8. #8
    Uber Raid King Computurd's Avatar
    Join Date
    Aug 2009
    Location
    Wichita, Ks
    Posts
    3,887
    Well guys there is hope in sight. Well, it was already in sight years ago, just not many people spreading the word.

    Newer RAID controllers, and even older ones, will just skip the sector during rebuild, and you lose that sector. So you lost some bit of info, and if its part of a larger file that could be bad of course. IF you are unlucky enough for it to be in the MFT it could really be suck too.

    However, we should not have forgotten the #1 rule...RAID IS NOT BACKUP.

    Keep multiple copies of your data, and keep an offsite copy of your data for the just-in-case. We now also have the options of DropBox, SkyDrive, and God only knows how many other cloud backup solutions that are very cheap. Hell, my ISP gives me 50GB of cloud storage for free. Use these for your files requiring the utmost data security.

    I personally keep a copy of everything bitlocker'ed at the in-laws, I used to keep a copy in a safe deposit box, but that got old and time consuming.

    Make sure to keep multiple copies, and when you run across that bad file from a rebuild you can just restore the file.

    Or better yet, dont even rebuild. Just wipe it and copy your backed up data onto the array.

    If you do run into an issue with an array and you must rebuild the first order of the day is to clone each and every disk in the array. Shut it down, take each disk and run a clone on it. If you do lose a sector that the controller leaves blank, thus ruining a file, you have another shot at recovery



    -----I need to update the RAID controllers in my sig
    Last edited by Computurd; 07-25-2014 at 05:29 AM.
    "Lurking" Since 1977


    Jesus Saves, God Backs-Up
    *I come to the news section to ban people, not read complaints.*-[XC]Gomeler
    Don't believe Squish, his hardware does control him!

  9. #9
    Xtreme Member DocGolem's Avatar
    Join Date
    Apr 2004
    Location
    Iowa
    Posts
    109
    Now Seagate has said they are coming out with 8TB drives! http://www.seagate.com/about/newsroo...ves-pr-master/
    i7 930 w/ Apogee XT
    3x2GB Mushkin Redline DDR3 2000
    eVGA E758
    eVGA GTX480 w/EK FC Block
    Corsair 850W

  10. #10
    Xtreme Member
    Join Date
    Feb 2012
    Location
    Perth, Australia
    Posts
    466
    Quote Originally Posted by knopflerbruce View Post
    My first thought when reading this is that the URE specs don't necessarily have to be realistic. From what I've seen they haven't changed for ages. That just sounds strange. It's an interesting topic, but it would be better if we had some hard facts regarding URE's rather than product specs.
    This is the problem right here. I have 8 x 2tb consumer drives at home. I bet if I ran a several disk test on these when I get home I wouldn't get a data read error. Given the reasoning in this thread, I should get at least a couple of errors if I did do this test, but I bet I won't.

    The reason is because URE's are NOT random errors. Errors in drives are caused by the same problems drives have experienced basically forever, which include failed or worn hardware, system noise, power failure.

    If you were to get a very large randomly selected collection of these consumer drives, you would be able to measure this the URE and maybe get close to this rate, and you would find that among your population, you would find most of the URE's are being caused by drives that are failing or have problems that are specific to that drive. Remove the failing drives from your collection, and URE rates would drop significantly.

    Additionally, all the URE rates are calculated mathematically, based on supposed mathematical models of off the constituent components. It isn't a completely useless value, but I wouldn't take it at face value as being correct either, as no drive manufacturer actually tests this properly anymore.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •