0

When a (Archlinux) system that has its root on RAID and LVM is misconfigured and does not boot, what steps are needed to recover it?

Tinco
  • 137
  • 7
  • Anyone can make a mistake but to make the same mistake three times takes an incredible amount of carelessness. – John Gardeniers Nov 13 '12 at 04:47
  • @JohnGardeniers ...or a lot of hard work and careful planning. The real trick is making it look like a different mistake, and someone else's fault so you can bill for all the extra hours, though. – HopelessN00b Nov 13 '12 at 15:54
  • @JohnGardeniers my apologies, I had misunderstood that this site is for people running servers, while it is for people running sites in a /professional/ setting. I am very careless with this server, and I should've posted my question on Linux&Unix instead of here :) Not sure why it's not a real question though. I still think it is useful information, even if it happens just once instead of three times. – Tinco Nov 14 '12 at 21:35
  • @Tinco, I suggest investing in a UPS to help eliminate this kind of problem. – John Gardeniers Nov 15 '12 at 00:09

1 Answers1

3

First, get a linux live usb stick, I use the debian one and boot from it.

If your server is under a staircase or some other hard to sit location do the following three steps:

1 Become root, type adduser myname and enter your login information.

2 Add the line myname ALL=(ALL) ALL to /etc/sudoers.

3 Type sshd to start the sshd daemon if it is not already running (it is on the debian stick).

Now I assume you are logged into your machine. Type sudo -s to become root.

To find your raid arrays and mount them perform the following two steps:

1 Type mdadm --examine --scan > /etc/mdadm/mdadm.conf to setup the configuration.

2 Type madam --assemble --scan to ready the devices.

Now to mount your LVM partitions do the following steps:

1 Type lvscan to find all your volumes.

2 Type vgchange -ay dirname where dirname is the directory of your volume groups. (for me /dev/data and /dev/array because lvscan shows /dev/data/home and /dev/array/root)

3 For each volume you can now make a directory in /mnt and mount the partition there by typing mount /dev/array/root /mnt/root when root is the name of the partition you want to mount and /dev/array/root the volume it is in.

Now type mkdir /mnt/boot and mount your boot partition there if you have your boot directory on a separate partition I have mine on a raid device too so I type mount /dev/md1 /mnt/root/boot.

Now we have all filesystems setup, it's time to chroot into our broken system. First change into your root directory like so cd /mnt/root. Then perform the following series of commands:

mount -t proc proc proc/
mount -t sysfs sys sys/
mount -o bind /dev dev/
mount -t devpts pts dev/pts/

Now it's time for the main event, type chroot . /bin/bash and you are back into your old system. You can repair it like normal, perhaps run pacman -Syu to install all updates, and don't forget to run mkinitcpio if your kernel changes!

Tinco
  • 137
  • 7