Page 2 of 2 FirstFirst 12
Results 26 to 32 of 32

Thread: Applications and Resources for Bit Error Recovery in Stored Data

  1. #26
    Xtreme Mentor
    Join Date
    Sep 2006
    Posts
    3,246
    Added HashCheck Shell Extension to OP.

    Nice little app.

    sha-1, md5, md4, crc-32
    [SIGPIC][/SIGPIC]

  2. #27
    Xtreme Mentor
    Join Date
    Sep 2006
    Posts
    3,246
    Adding ICE ECC:

    http://www.ice-graphics.com/ICEECC/IndexE.html


    I find quickpar has a memory leak so I can no longer recommend it. ICE is great. It's the main app I use now for ECC data protection.
    The quickpar problem might only be on 64 bit Vista, I don't know. Since I have a better alternative I'm not motivated to test it.
    Last edited by Speederlander; 06-07-2009 at 07:45 AM.
    [SIGPIC][/SIGPIC]

  3. #28
    Xtreme Addict
    Join Date
    Jul 2006
    Posts
    1,124
    Just to add a couple comments here as things have progressed a little at least for the Linux crowd. DIF/DIX has been added into the 2.6.30+ kernels which give some additional support for data integrity protections (520 byte sectors (512+8)) which still unfortunately are only helpful to enterprise disc users (SAS/SCSI/FC) not general consumer interface types (there was a proposel to the T13 working committee to add 520-528 byte sector support but haven't seen much traction).

    http://ols.fedoraproject.org/OLS/Rep...en-reprint.pdf
    http://oss.oracle.com/projects/data-.../linux-hba.pdf
    http://oss.oracle.com/projects/data-...008-slides.pdf

    Sadly, I have not seen raid array controllers support this yet, though if enough people put some pressure on them it may yet come.

    Now we still need the 10^16 UER rates desperately for large drives.
    Last edited by stevecs; 09-16-2009 at 01:45 PM.

    |.Server/Storage System.............|.Gaming/Work System..............................|.Sundry...... ............|
    |.Supermico X8DTH-6f................|.Asus Z9PE-D8 WS.................................|.HP LP3065 30"LCD Monitor.|
    |.(2) Xeon X5690....................|.2xE5-2643 v2....................................|.Mino lta magicolor 7450..|
    |.(192GB) Samsung PC10600 ECC.......|.2xEVGA nVidia GTX670 4GB........................|.Nikon coolscan 9000......|
    |.800W Redundant PSU................|.(8x8GB) Kingston DDR3-1600 ECC..................|.Quantum LTO-4HH..........|
    |.NEC Slimline DVD RW DL............|.Corsair AX1200..................................|........ .................|
    |.(..6) LSI 9200-8e HBAs............|.Lite-On iHBS112.................................|.Dell D820 Laptop.........|
    |.(..8) ST9300653SS (300GB) (RAID0).|.PA120.3, Apogee, MCW N&S bridge.................|...2.33Ghz; 8GB Ram;......|
    |.(112) ST2000DL003 (2TB) (RAIDZ2)..|.(1) Areca ARC1880ix-8 512MiB Cache..............|...DVDRW; 128GB SSD.......|
    |.(..2) ST9146803SS (146GB) (RAID-1)|.(8) Intel SSD 520 240GB (RAID6).................|...Ubuntu 12.04 64bit.....|
    |.Ubuntu 12.04 64bit Server.........|.Windows 7 x64 Pro...............................|............... ..........|

  4. #29
    Xtreme Addict
    Join Date
    Jul 2006
    Posts
    1,124
    At the beginning of the year I threw together a fast script that I posted here for others to check their arrays for bit errors in files. Since then I have both increased the number of arrays here (eight 8-way 1TB drive RAID-6's on the test server) and constrained the array sizes to cut down on UBE percentages. With the larger sampling sizes I see an average of ~20 bit errors in files per month (corrupted/changed files). In none of the cases did the raid controller discover any errors from a raid level on down (all silent errors). In light of this I pretty much got fed up just identifying the files that had issues and doing manual restores and went to modify the old script to add in par2 support for entire mount points. PAR2 is a pretty bad implementation of reed-solomon (inefficient in compute cycles, and program interface is very bad for handling multiple (i.e. thousands/millions) of files. It needs a major re-write or if someone knows of a different program?

    Anyway, I managed to force it into a scripted state that passably works and it's attached. It should be able to run on most *nix (linux and bsd like systems (solaris, irix, et al)) as long as the programs are installed (sha1/md5/par2/et al). This would include cygwin under windows and similar. I've tested it here under linux with ~2,500,000 files of various sizes (32KiB up to ~120GiB). It's multi-threaded so the more cores you have (assuming you don't have an I/O bottleneck) the better.

    Anyway, in hopes that it can help someone else who's also pulling their hair out and may not have the time to throw their own utility together. Especially since BTRFS/ZFS is at least several years away from a production standard (unless you're running solaris then ZFS is good). And DIX/DIF support under T13 (SATA) is equally long off this is the only real option at this point if data integrity is a major concern.

    Like always, free to use/modify/abuse.
    Attached Files Attached Files
    Last edited by stevecs; 11-02-2009 at 02:27 PM.

    |.Server/Storage System.............|.Gaming/Work System..............................|.Sundry...... ............|
    |.Supermico X8DTH-6f................|.Asus Z9PE-D8 WS.................................|.HP LP3065 30"LCD Monitor.|
    |.(2) Xeon X5690....................|.2xE5-2643 v2....................................|.Mino lta magicolor 7450..|
    |.(192GB) Samsung PC10600 ECC.......|.2xEVGA nVidia GTX670 4GB........................|.Nikon coolscan 9000......|
    |.800W Redundant PSU................|.(8x8GB) Kingston DDR3-1600 ECC..................|.Quantum LTO-4HH..........|
    |.NEC Slimline DVD RW DL............|.Corsair AX1200..................................|........ .................|
    |.(..6) LSI 9200-8e HBAs............|.Lite-On iHBS112.................................|.Dell D820 Laptop.........|
    |.(..8) ST9300653SS (300GB) (RAID0).|.PA120.3, Apogee, MCW N&S bridge.................|...2.33Ghz; 8GB Ram;......|
    |.(112) ST2000DL003 (2TB) (RAIDZ2)..|.(1) Areca ARC1880ix-8 512MiB Cache..............|...DVDRW; 128GB SSD.......|
    |.(..2) ST9146803SS (146GB) (RAID-1)|.(8) Intel SSD 520 240GB (RAID6).................|...Ubuntu 12.04 64bit.....|
    |.Ubuntu 12.04 64bit Server.........|.Windows 7 x64 Pro...............................|............... ..........|

  5. #30
    Xtreme Mentor
    Join Date
    Sep 2006
    Posts
    3,246
    That's 20 bit errors total for all of those arrays, yes? What drives? I assume they are all 10^15 drives?
    [SIGPIC][/SIGPIC]

  6. #31
    Xtreme Addict
    Join Date
    Jul 2006
    Posts
    1,124
    At least 20 bit errors, actually it's 20 files that would not match SHA1/MD5 so there is a possibility that there was more than 1 bit error per file but at least 20. PER MONTH. All drives are 10^15 which just really covers UBE's not items like torn writes, lost writes, wild writes, and other types of drive low level issues. Drives in the tests here are the ones in my .sig (ST31000340NS)'s On the sas drives (1/15 the size) I have seen zero bit errors over the 8 month period where the testing has overlapped.

    |.Server/Storage System.............|.Gaming/Work System..............................|.Sundry...... ............|
    |.Supermico X8DTH-6f................|.Asus Z9PE-D8 WS.................................|.HP LP3065 30"LCD Monitor.|
    |.(2) Xeon X5690....................|.2xE5-2643 v2....................................|.Mino lta magicolor 7450..|
    |.(192GB) Samsung PC10600 ECC.......|.2xEVGA nVidia GTX670 4GB........................|.Nikon coolscan 9000......|
    |.800W Redundant PSU................|.(8x8GB) Kingston DDR3-1600 ECC..................|.Quantum LTO-4HH..........|
    |.NEC Slimline DVD RW DL............|.Corsair AX1200..................................|........ .................|
    |.(..6) LSI 9200-8e HBAs............|.Lite-On iHBS112.................................|.Dell D820 Laptop.........|
    |.(..8) ST9300653SS (300GB) (RAID0).|.PA120.3, Apogee, MCW N&S bridge.................|...2.33Ghz; 8GB Ram;......|
    |.(112) ST2000DL003 (2TB) (RAIDZ2)..|.(1) Areca ARC1880ix-8 512MiB Cache..............|...DVDRW; 128GB SSD.......|
    |.(..2) ST9146803SS (146GB) (RAID-1)|.(8) Intel SSD 520 240GB (RAID6).................|...Ubuntu 12.04 64bit.....|
    |.Ubuntu 12.04 64bit Server.........|.Windows 7 x64 Pro...............................|............... ..........|

  7. #32
    Xtreme Member
    Join Date
    Oct 2006
    Posts
    408
    Thank you sir!
    say it ain't so Joe............SEO Services SEO Business

Page 2 of 2 FirstFirst 12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •