3

While running a Hadoop cluster that uses HDFS (so data is already replicated 3x) we experience often issues with ext4 because some bad sectors (the filesystem is unable to read or write to them) causing ext4 to remount the filesystem read-only..

So far so good and we working on replacing the disks but I've stumbled about btrfs and it's metadata duplication and I'm interested how btrfs does react in such a situation?

Data errors do not matter for us, as the data is already checksummed and replicated by HDFS but a more robust metadata handling (e.g. if metadata can't be written in one place the duplicated metadata is used) would be a benefit because in theory the read-only remounts and required fsck should not happen that often if we would switch to btrfs...

So is here anyone running anyone btrfs without raid on desktop-hdds and can tell me how resilient the filesystem is against medium errors?

E.g. does the metadata duplication on a single disk is used to repair broken metadata or does the filesystem fail anyway?

kei1aeh5quahQu4U
  • 445
  • 5
  • 22
  • If you're only using a single disk, you're really losing most of the benefits of btrfs, and not really gaining any fault tolerance. Go buy more disks. – Michael Hampton Feb 17 '14 at 15:45
  • 1
    This place is frustrating sometimes. Did you read my question? RAID does not make sense for Hadoop - so JBOD is the only choice. It's a question about that exact problem. Metadata replication on a single disk or multiple disks and how this works in practice. I know that I can buy more disks. Sorry... – kei1aeh5quahQu4U Feb 18 '14 at 05:31

1 Answers1

1

I am definitely NO expert and have very little experience on filesystem in general. So, take what I write with pinch(or handful of salt) :)

Now disclaimer aside: Btrfs (to the best of my knowledge ) is not YET fault tolerant and it needs serious working in that department.If my memory serves right ZFS should serve your requirement better. I had myself been considering btrfs but I am not yet keen on using it. Incidently opensuse (I think) provide support for it, so maybe you find some info there.

Please do update, if you find solution elsewhere.

Hope, I was of some help.

http://en.wikipedia.org/wiki/ZFS

For ZFS, data integrity is achieved by using a (Fletcher-based) checksum or a (SHA-256) hash throughout the file system tree.[17] Each block of data is checksummed and the checksum value is then saved in the pointer to that block—rather than at the actual block itself. Next, the block pointer is checksummed, with the value being saved at its pointer. This checksumming continues all the way up the file system's data hierarchy to the root node, which is also checksummed, thus creating a Merkle tree.[17] In-flight data corruption or phantom reads/writes (the data written/read checksums correctly but is actually wrong) are undetectable by most filesystems as they store the checksum with the data. ZFS stores the checksum of each block in its parent block pointer so the entire pool self-validates.[18]

When a block is accessed, regardless of whether it is data or meta-data, its checksum is calculated and compared with the stored checksum value of what it "should" be. If the checksums match, the data are passed up the programming stack to the process that asked for it. If the values do not match, then ZFS can heal the data if the storage pool has redundancy via ZFS mirroring or RAID.[19] If the storage pool consists of a single disk, it is possible to provide such redundancy by specifying "copies=2" (or "copies=3"), which means that data will be stored twice (thrice) on the disk, effectively halving (or, for "copies=3", reducing to one third) the storage capacity of the disk.[20] If redundancy exists, ZFS will fetch a copy of the data (or recreate it via a RAID recovery mechanism), and recalculate the checksum—ideally resulting in the reproduction of the originally expected value. If the data passes this integrity check, the system can then update the faulty copy with known-good data so that redundancy can be restored.

vedic
  • 186
  • 3
  • I had made it clear I am no expert and sharing my thoughts so downvote doesnot make sense and I had to write as answer because I couldnt comment – vedic Feb 18 '14 at 06:20
  • your answer is actually pretty good, I did not downvote you! not sure what is going on here... – kei1aeh5quahQu4U Feb 18 '14 at 13:16
  • 3
    Half your answer is spreading FUD about BTRFS with no evidence of actual knowledge, and the other half is a copy-and-paste from wikipedia. – sciurus Feb 18 '14 at 15:55
  • oh really! I have USED BTRFS and even though I have battery backup the filesystem was corrupted on power failure and system was rendered useless. OS was Opensuse. I was lucky that system was NOT mission critical but I decided to try btrfs when it matures enough to be considered for mission critical stuff and yes 2nd half is wikipedia stuff I have given link also, so it shows that I do loads of research and this place is for the same. I would have appreciated you providing input rather than criticizing on some wild assumption. – vedic Feb 18 '14 at 16:07
  • That's generous of you thank you, and if possible, please do update on any solution you find and a link http://docs.oracle.com/cd/E37670_01/E37355/html/ol_create_btrfs.html hope it helps – vedic Feb 24 '14 at 11:02
  • I'm testing it right now on a few broken drives so I guess in a few weeks I can report back... – kei1aeh5quahQu4U Feb 24 '14 at 21:47
  • 1
    I tested ZFS and it works great for these kinds of problems. I've found a paper on ZFS and disk problems: http://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf – kei1aeh5quahQu4U Apr 26 '14 at 12:14