1

I have a raid5 NAS on Open Media Vault, with 3x 2To disks.

A few days ago I've inserted a spare disk from another raid. I did not see it at the beginning, but I think that changed disks order. Then I run cfdisk to make a new volume, maybe on the wrong /dev/. I think, not sure of that.

Nothing was wrong until the system got rebooted.

Now I'm stuck at boot with a very bad message

journalctl :

Dec 27 20:44:13 pacem systemd-fsck[362]: /dev/md1: One or more block group descriptor checksums are invalid.  FIXED.
Dec 27 20:44:13 pacem systemd-fsck[362]: /dev/md1: Group descriptor 0 checksum is 0x0000, should be 0xbd42.
Dec 27 20:44:13 pacem systemd-fsck[362]: /dev/md1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Dec 27 20:44:13 pacem systemd-fsck[362]: (i.e., without -a or -p options)

Raid seems to be fine

cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] 
md1 : active raid5 sdd1[0] sdc1[3] sdb1[1]
  3907026624 blocks super 1.0 level 5, 32k chunk, algorithm 2 [3/3] [UUU]
  bitmap: 0/15 pages [0KB], 65536KB chunk

Also I've check dumpe2fs /dev/md1 | grep -i superblock It says `ext2fs_read_bb_inode: A block group is missing at inode table' but I see a list of superblocks backup.

Is there any chance to restore this filesystem ? What's the best thing to try first ?

Barium Scoorge
  • 135
  • 1
  • 6
  • I'm currently copying 2 of 3 array devices to images on a big disk using dd then I'll trying to assemble md array from images (with a missing disk) , then try to fsck and recover ext4.. – Barium Scoorge Dec 28 '19 at 17:06
  • used images to assemble a raid. after fdisk has done its job, it seems that my volume is empty... but not ! my main directories are listed in subfolders of lost+found. – Barium Scoorge Dec 29 '19 at 18:14

1 Answers1

0

As I successfully retrieved my data, I will answer my own question, this maybe could be useful to someone.

Warning :

  1. I've been confused by the fact that when "repaired" after fsck, root directories of my drive where "hidden" inside lost+found directory
  2. I will describe here how I managed to "safely" try to repair my faulty ext4
  3. As always : Keep backup of ALL data. My partial backup was a pain, I did not expect that..
  4. Hot swap drive bay was a real advantage in my situation : no need to reboot when swapping between drives

Prepare

  • I've bought a 6TB drive (named backup)
  • Once backup drive in place and formatted (etx4), I've duplicated 2 of 3 of the raid5 drives, using dd :dd if=/dev/sdb of=/mnt/backup/sdb1.img ; dd if=/dev/sdc of=/mnt/backup/sdc1.img. Took 10 hours @ 110mb/s
  • Switch to another PC with backup drive
  • Using loopback devices, created devices with previous images, as a sort of virtual drive
  • losetup -f (initiate /dev/loopX if missing)
  • losetup /dev/loop0 /mnt/backup/sdb1.img creates /dev/loop0
  • losetup /dev/loop1 /mnt/backup/sdc1.img creates /dev/loop1
  • Now, starting a new raid5, with only 2 drives mdadm --assemble --run /dev/md1 /dev/loop0 /dev/loop1. It should be marked with a [UU_] when display cat /prod/mdstat
  • Note that later, once /dev/loopX is created, mdadm might create/dev/md127on his own. you can stop it withmdadm --stop /dev/md127to create your own/dev/mdX`

Repair

  • Now it's time to try repairing filesystem, since trying to mount /dev/md0 shows a Group descriptor 0 checksum is 0x0000, should be 0xbd42 ...
  • run fsck.ext4 /dev/md0
  • To answer all questions with yes, hit key a
  • Takes a few hours, 2 cycles of fsck maybe needed (when repaired, fsck is restarted from the begining)

Mount and backup

  • Then I could run mount /dev/md0 /mnt/data
  • But /mnt/data is empty, except lost+found/ !
  • It seems that the root directories are somewhat correupted, and then all my data is stored inside lost+found/#17xxxxxx folders !
  • ls -la /mnt/data/lost+found/\#17* to display all content
  • .. Then backup again to my new raid (bought in between 3x 4TB)

Resources :

https://askubuntu.com/questions/322070/how-can-i-fix-mounting-my-data-drive-after-a-crash

https://askubuntu.com/questions/19901/how-to-make-a-disk-image-and-restore-from-it-later

https://superuser.com/questions/962395/assemble-3-drive-software-raid5-with-one-disk-missing

Barium Scoorge
  • 135
  • 1
  • 6