bit error = lost file, reduce BER how?

[stevecs]

Yeah, the INFO's you can ignore they are just telling you what functions are being entered (it /IS/ only an after-dinner first pass. :P) The ERROR one basically is saying that there is no previous output to compare the file system to in the first draft it is set up (hard-coded) to write a file called md5sum.<yearmonth>01 (I must have forgotten to fix that) was going to just search the $HASHDIR for the newest output and use that. Anyway, I ran it against about 900,000 files here both unicode-16 and normal file names (multi-byte characters) so that's clean at least and it does multi-threading (which was the main thing I was going for).

create the directory /var/log/md5sum then run it first in create mode:
md5check.sh <directory> create
then run it in check mode
md5check.sh <directory> check


I've been e-mailing Eric Gerbier (writer of afick http://afick.sourceforge.net/) to see if we can merge ideas. Mine above works well with large files (100MiB+) (was able to push 800MiB/s throughput on a 2.4Ghz quad core which fully saturated the areca 1280ML) but with small files the spawning of the md5sum is a killer. Eric uses perl which has md5/sha1 already built in (no spawning penalty) however he doesn't have any multi-threading (so limited to about 200-250MiB/sec max on a 2.4Ghz core2). He also is already using GDBM, which is what I was going to do next in mine.