Applications and Resources for Bit Error Recovery in Stored Data

[Speederlander]

Added HashCheck Shell Extension to OP.

Nice little app.

sha-1, md5, md4, crc-32

[Speederlander]

Adding ICE ECC:

http://www.ice-graphics.com/ICEECC/IndexE.html


I find quickpar has a memory leak so I can no longer recommend it. ICE is great. It's the main app I use now for ECC data protection.
The quickpar problem might only be on 64 bit Vista, I don't know. Since I have a better alternative I'm not motivated to test it.

[stevecs]

Just to add a couple comments here as things have progressed a little at least for the Linux crowd. DIF/DIX has been added into the 2.6.30+ kernels which give some additional support for data integrity protections (520 byte sectors (512+8)) which still unfortunately are only helpful to enterprise disc users (SAS/SCSI/FC) not general consumer interface types (there was a proposel to the T13 working committee to add 520-528 byte sector support but haven't seen much traction).

http://ols.fedoraproject.org/OLS/Rep...en-reprint.pdf
http://oss.oracle.com/projects/data-.../linux-hba.pdf
http://oss.oracle.com/projects/data-...008-slides.pdf

Sadly, I have not seen raid array controllers support this yet, though if enough people put some pressure on them it may yet come.

Now we still need the 10^16 UER rates desperately for large drives.

[stevecs]

At the beginning of the year I threw together a fast script that I posted here for others to check their arrays for bit errors in files. Since then I have both increased the number of arrays here (eight 8-way 1TB drive RAID-6's on the test server) and constrained the array sizes to cut down on UBE percentages. With the larger sampling sizes I see an average of ~20 bit errors in files per month (corrupted/changed files). In none of the cases did the raid controller discover any errors from a raid level on down (all silent errors). In light of this I pretty much got fed up just identifying the files that had issues and doing manual restores and went to modify the old script to add in par2 support for entire mount points. PAR2 is a pretty bad implementation of reed-solomon (inefficient in compute cycles, and program interface is very bad for handling multiple (i.e. thousands/millions) of files. It needs a major re-write or if someone knows of a different program?

Anyway, I managed to force it into a scripted state that passably works and it's attached. It should be able to run on most *nix (linux and bsd like systems (solaris, irix, et al)) as long as the programs are installed (sha1/md5/par2/et al). This would include cygwin under windows and similar. I've tested it here under linux with ~2,500,000 files of various sizes (32KiB up to ~120GiB). It's multi-threaded so the more cores you have (assuming you don't have an I/O bottleneck) the better.

Anyway, in hopes that it can help someone else who's also pulling their hair out and may not have the time to throw their own utility together. Especially since BTRFS/ZFS is at least several years away from a production standard (unless you're running solaris then ZFS is good). And DIX/DIF support under T13 (SATA) is equally long off this is the only real option at this point if data integrity is a major concern.

Like always, free to use/modify/abuse.

[Speederlander]

That's 20 bit errors total for all of those arrays, yes? What drives? I assume they are all 10^15 drives?

[stevecs]

At least 20 bit errors, actually it's 20 files that would not match SHA1/MD5 so there is a possibility that there was more than 1 bit error per file but at least 20. PER MONTH. All drives are 10^15 which just really covers UBE's not items like torn writes, lost writes, wild writes, and other types of drive low level issues. Drives in the tests here are the ones in my .sig (ST31000340NS)'s On the sas drives (1/15 the size) I have seen zero bit errors over the 8 month period where the testing has overlapped.

[swiftex]

Thank you sir!