Applications and Resources for Bit Error Recovery in Stored Data

[IanB]

I'd agree that PAR2 in its current implementation would not be a perfect solution for continual OS-level use. The reason being that every time a file is changed, even just a single byte, the small PAR2 file would have to be rebuilt from scratch, with the checksums of every file in the folder (not just the one that had changed) and then, even more wastefully in terms of processor usage and time, any repair files would have to be recreated from the entire folder set. The disk overhead and CPU usage would be continual and enormous in light, normal file activity, let alone any active file processing.

But for archived files where you rarely change the contents of folders and simply want to guarantee their long-term longevity, PAR2 is ideal. It's particularly handy for filling any spare contents of archived DVDs with repair files so that you can guard against bit-faults due to "fade" over time. What hasn't yet been done I guess is hooking up a repair facility to an automatic timed mechanism for checking the contents to remove the need for manual checking/repair. Hardly difficult. QuickPar was designed from the outset for manual use with PARed Usenet files, which is why it never had such automatic facilities built in.

That basically means that you will be on average using up to 1.2 - 1.5x the amount of dead disk space for online applications.

I'm not sure where you get that excessive figure from. You can create as many or as few repair blocks as you need. A repair set can contain as many as 32768 virtual blocks (that's the limitation in the maths) so the size of the smallest repair file is 1/32768 (0.003%) of the total fileset size plus the miniscule packet overhead. That would give you one repair block, which can be reused as many times as you need whenever a bit error (EDIT: or multiple errors in a single virtual block) occurs in the fileset. More redundancy (just like RAID 6 over RAID 5) means more repair blocks, able to repair more simultaneous bit errors (EDIT: where they occur in different virtual fileset blocks). But 0.003% is a LONG way from 20% to 50%, that is a HUGE amount of redundancy that bears no relation to the likelihood of bit errors in disk storage. Even for Usenet where some servers were poor at propagating without errors, 10% PAR2 cover was generous back when I was doing that...