0

I have 2 physical disks in linux RAID1, I am not sure what happened, I think someone who should not have been in the system partitioned the md0 by mistake while the system was running while trying to add a new disk. Regardless, on reboot, it went to grub. I have spent days trying to fix this with systemrescueCD, and stopped the array which showed "UU" before I stopped it, so it should be good, but I am having an issue with re-assembling it and getting it back to normal operation. I am not sure what to do. It shows FAT32 filesystem, but also ext2, and shows backup superblocks, but also says no superblock exists. Thank you in advance for your help!!

SCREENSHOTS BELOW

  • Here are the disks
    Here are the disks]1

  • Superblock backups appear to exists
    Superblock backups appear to exist

  • Drive and Filesystem structure
    Drive and filesystem structure

  • Can't use a backup superblock, bad magic number
    Can't use a backup superblock, bad magic number

  • Can't force assembly either
    Can't force assembly either

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
Rob
  • 1
  • 1
  • 2
    You have already destroyed your data on both disks. It's time to restore from backup. – Michael Hampton Jul 27 '21 at 22:21
  • Can you explain briefly the reasoning behind this? Is it because the md0 was partitioned? – Rob Jul 27 '21 at 22:55
  • Neither partition has an mdraid superblock on it any more, according to your screenshot from attempting to assemble it, and also from your lsblk output. That means it was overwritten on both disks. You are going to have a hell of a time reassembling the array without its metadata. There might be some black magic that will do it but it's beyond me. But even then, the filesystem seems to be gone as well. Restoring from backup would be much easier and would actually work. – Michael Hampton Jul 27 '21 at 22:59
  • I was looking to restore a superblock backup from the ones listed, but it seems that for some reason it isn't working. I wouldn't be going through all this hassle, but just nearing the end of an openstack deployment someone screwed up the controller node, and it hadn't been backed up yet as it was in the final stage. So I am trying all options here. – Rob Jul 27 '21 at 23:06
  • 1
    You're confusing the filesystem superblock with the mdraid superblock. The latter is _gone_ on both disks, and you can't assemble without it. No backup? Time to start over, then. – Michael Hampton Jul 27 '21 at 23:12
  • Live with the fact that the raid is DEAD. In case of no backup - either try with a Data Restore Company and pay thousands or start over. – djdomi Jul 28 '21 at 12:50
  • I would like to point out you're performing operations on the disk, not the partition /dev/sda2 and /dev/sdb2. – Halfgaar Jul 28 '21 at 14:44
  • When you ran mkfs on both sda and sdb you fortunately used the -n switch, maybe you have not wiped your data. Usually before starting to rescue data it is a really good idea to make a disk image file and work on copies of that file. Trying to rescue on the disk might cause unnecessary data loss if you make a mistake. – Henrik Carlqvist Jul 28 '21 at 15:29

1 Answers1

0

I doubt it's going to be of help in your case, but I would like to point out the theoretical case of recovering the file system inside the array. It's mirror RAID, so you don't need the MD system to access the file system. You just need to know where it is.

Depending on whether you have superblock version 0.9, 1.0, 1.1 or 1.2, the superblock is either at the beginning or the end. If it's at the end, a component partition like /dev/sda2 should just be mountable as file system. If it's at the beginning, you can create a loop device with offset using losetup. (note: never re-assemble an array you do this on, because the drives are out of sync)

For instance, losetup --find --verbose --offset 4096 /dev/sda2 will create a loop device on /dev/sda2, starting 4096 bytes into the partition. This may work for MD superblock 1.2. Disclaimer: untested command, but the theory is sound.

You could then dd/pv it into a new MD array:

  • create new MD array, device /dev/mdXYZ:
  • pv /dev/loopX > /dev/mdXYZ

Of course, there are many more things to do to actually boot from it, or in some other way get it to work in another machine, but that's out of scope here.

Halfgaar
  • 8,084
  • 6
  • 45
  • 86
  • Thank you very much. I would have tried this but decided to already to start fresh. Thanks everyone for the advice. I learned a lot, ie., don't let unqualified people mess with anything, and if I had it deployed in HA, I wouldn't be facing this issue at all. I appreciate everyone's comments! – Rob Jul 28 '21 at 22:19