1

Are there any good strategies for proactively detecting data corruption in a Greyhole pool.

Suppose the following chain of events happens.

c:\> copy swiss_bank_account.txt \\greyhole\safe_documents

Greyhole does its thing and replaces:
  safe_documents/swiss_bank_account.txt with -> /mnt/pool1/safe_documents/swiss_bank_account.txt
and creates a backup file:
  /mnt/pool2/safe_documents/swiss_bank_account.txt

/mnt/pool2 suffers a random failure, corrupting swiss_bank_account.txt - It goes un-noticed because it's the secondary.
/mnt/pool1 suffers a random failure - Crap... now both my redundant copies are corrupt.

What are good strategies for proactively detecting corruption in a JBOD-style duplication array like Greyhole?

Unless I'm mistaken, even 3-way replication isn't fool-proof. In the event of a catastrophically failed drive, you could only detect, not resolve discrepancies between the 2 surviving copies.

The viable systems I can think of are:

  1. 3-way replication across a checksummed file system, like btrfs.
  2. 3-way replication and hope all your failures are uncorrelated.
  3. Chron-job application of parity tools.
  4. Hooking Greyhole to run parity tools on write.
  5. Chron-job scans for data agreement.

Other than option 1, and 2, all of these options seem like more work than I'd like to put into my home server. Does anyone have any suggestions?

Tom O'Connor
  • 27,480
  • 10
  • 73
  • 148
Kennet Belenky
  • 121
  • 3
  • Greyhole is by it's own definition a product for home users, not professionals. Therefore, it's off-topic here. Please see our [FAQ]. – Sven May 11 '12 at 20:36
  • 1
    @SvenW. Fair enough, however I only really mentioned Greyhole to add context. The question could be posed for any situation where you have to manage mirrored arrays of files. – Kennet Belenky May 11 '12 at 20:38
  • 1
    Why are you storing a bank account number in plain text? – Jacob May 11 '12 at 20:39
  • @Jacob. I assume you're joking. The files in question are actually family photos. However, money is what is universally considered "important data." – Kennet Belenky May 11 '12 at 20:42

2 Answers2

2

You're assuming that corruption will creep in silently, which is a form of data-integrity loss that both file-system and storage hardware developers actively work to avoid. Chances are real good that the block-driver will notice something went wrong with a read/write and yell about it, at which point it's up to the higher level software (Greyhole) to handle the fault. Or if the block-driver doesn't notice it, the filesystem driver will.

The case I think you're worried about is if cosmic rays or something twiddle the bits for one file on different devices, how does Greyhole handle that problem?

You're pretty much screwed, so you should go for a 3x redundancy if you're worried about it. However, the chances of three devices going bad at the same time are a lot less than only one going bad so it's a pretty far edge case.

sysadmin1138
  • 133,124
  • 18
  • 176
  • 300
  • I'm not worried about a triple failure. I'd be more worried about a double failure with no good way to figure out which surviving copy is intact. It sounds like you're saying that I can safely treat 3-way replication as 3-way replication with checksumming, even without a paranoid FS like brtfs or ZFS. – Kennet Belenky May 11 '12 at 20:45
  • @KennetBelenky It's all about risk-management. The type of problem that only a checksumming FS would catch is actually pretty rare. For a 2-device mirror, data will either be there in its entirety and retrievable w/o error, or there will be errors on one copy, which shows which device should be trusted. Checksumming only helps if data has been *randomized* in a way that the storage hardware and OS drivers won't notice and yet both copies are readable w/o fault. – sysadmin1138 May 11 '12 at 20:49
  • The data-randomization thing really is rare, *especially* in home systems where the storage path is generally really short. What it protects against best is data being randomized *in transit* to the storage device, which can occasionally happen if you're moving data over Fibre Channel, Infiniband, or iSCSI over very long runs (like over 10km). SATA on your motherboard is about as short a path as it gets. – sysadmin1138 May 11 '12 at 20:53
  • Ok, good to know. I'm an SWE, not an SRE, so all this stuff that happens in the physical world is very foreign to me. – Kennet Belenky May 11 '12 at 20:57
2

Answering with a Greyhole-specific solution: use the --checksums option for --fsck:

-k, --checksums
      Read  ALL  files  in  your  storage  pool,  and  check  that file copies 
      are identical. This should identify any problem you might have with your 
      file-systems.
      NOTE: this can take a LONG time to complete, since it will read everything
      from all your drives!

You'll want to make sure your server can send emails, and to use the --email-report option at the same time, to receive a report once it completes. (The report is also saved to disk, if you prefer that. In /usr/share/greyhole/ I think...)

Guillaume Boudreau
  • 674
  • 1
  • 6
  • 13