5

My question is, how can I detect silent data corruption at my home PC (Windows + Linux). I do rsync like automatic backups (with version history). But how do I know in a few months/years that a long unused file (like a thesis or my banking history) is still intact? Maybe I am just backing up a corrupted file.

If it was only one file I could check every few weeks if it still works (hash or just opening it), but doing it on hundred of files is mundane. And even I find out that a file is corrupted, how do I know, when it was corrupted?

At this point, I need to find the date in my backup at which the file was still OK. If there is a working copy in it! and did not get corrupted as well.

Cern study on data integrity

Reg Edit
  • 4,671
  • 2
  • 17
  • 35
David Stark
  • 53
  • 1
  • 4
  • Firstly, you can't correct corruption unless it's written in a way to store some sort of redundancy. Secondly, you'll have to write a hash for each file, and compare that before each synchronisation. –  Jul 14 '12 at 19:53
  • To verify files you will basically need a way to hash them (MD5 or SHA1 should do). Create/update a hash whenever you save the file and periodically recheck them. This wouldn't be too hard to script, at least for a small number of files. – Renan Jul 14 '12 at 19:57
  • Yeah I think correction of corruption is only possible if I find a working backup or if I have correct parity data... – David Stark Jul 18 '12 at 16:45

1 Answers1

2

If you want to correct those lost datas, you have to use some sort of data duplication. Either RAID, either backups.
Backups are a prefered way because they protect you against erased files, filesystem crashes, etc. A RAID will only protect against partial and complete disk failure. You have to look at the system log to see any failure.
With Linux you can use ZFS. Every bunch of data have its own checksum and the filesystem driver take care of that, with live correction. I think this is currently the best method.

How to detect them ?
Well, with Linux it is very simple: appart from ZFS, just use software RAID and the default configuration does a data scrubing each month (Debian and Ubuntu, probably same thing for others).
With Windows... you have to use some tricks. For example, reading each file once a month (with a script) and looking into the system log if anything is reported.
If you have a hardawre RAID controler (not faike RAID, a real RAID card), most of them include the software to do the automatic scrubing.

Another method is to store files checksum, and regularly check back files against the checksums. Tripwire does this for Unix. You have to support the fact that a modified file have a different checksum. I don't know any software for Windows, but other people could.

Gregory MOUSSAT
  • 901
  • 4
  • 17
  • 39