unattended recovery from image

Question

I've got quite a few machines which sometimes need to be restored from an image. There's a slight problem that those machines are at different locations, don't have keyboards and sometimes don't even have displays next to them, so it's required that the recovery is done in a completely unattended way.

Some details: the machines are not servers. They have AMT, but nothing more. I've read that Ghost has something called LightsOut, but it requires installation on the boxes, which is out of question.

My ideal solution would a piece of software that could make an image of a whole disk (the disks are the same everywhere; there are multiple partitions, but layout is also the same on all boxes) and make a bootable recovery iso suitable for booting from USB flash drive which doesn't ask any questions apart from "remove the recovery disk and reboot me please". The idea is that non-technical personell should be able to perform the recovery with only remote assistance from the tech people - AMT will be of help here.

I've been toying with the idea of adapting SystemRescueCD - boot, ssh, restore disk - but I'd much prefer a complete, existing solution. Is there something that does what I want?

Answer 1

This question periodically pops up, but I think from your description your situation is a little different in that you have remote hands available. Usually the answer is "you're rolling dice doing this because there's no guarantee the OS will work properly without peripherals, if something comes up with errors no one can help, you'll risk locking yourself out remotely..."

I don't know what AMT is. I don't know what operating system you're running. If it's UNIX based, you could do something like running DD to image the disk to a file, then returning to that image is a matter of reverse-dd'ing the image back to the drive with the drive unmounted and your system running entirely from memory.

In that case you could have a procedure written up to follow. I've recovered some disks this way using Rescue Is Possible Linux (RIP); boot it up, log in as root, then set a password and run /usr/sbin/sshd and get the IP (run netconfig if it doesn't have one already) and then you can mount the remote share and copy over (or mount the USB device).

I've read of recoveries remotely done by unmounting the filesystem and, with the OS running in memory, doing this recovery. But be aware that it is very risky. A router goes pop, a network card gets reset, if your disk has an issue that means it's spitting errors and you don't see it...you could fubar hours of work and lose complete access to the machine anyway.

Plus if the disk is being replace your image may not entirely work properly, as the sector/head/cylinder count may be wonky and DD is going to have a copy of your original MBR. Many imaging utilities (dd, partimage, gparted, etc.) will have trouble from that.

Options to consider...restructure those systems to run from memory and mount a hard disk as a data storage area, much like Devil Linux (I believe it's called). You can create a Squid server or mail server or whatnot with that distro that runs from CD, and it mounts a local disk (if you need it) just for data. If someone cracks the system they can't install their own binaries because CD's are read only. Makes updates as easy as using a new CD and rebooting (configs can be kept on read-only media or on USB drives holding just config files, easily backed up).

You could also consider running the OS from an external drive (if it's Linux-based), or running from removable drives. Then you can upgrade/update/restore and just ship the drive to the remote location, and if it's a swappable disk the end user just needs to slide one out and pop in the new one.

No matter what, you must test the procedure. It's way too easy to come up with "in theory" and then when you actually have to restore have it fall flat, and discovering that when working with a user that can't tell a NIC from a hard disk on the other side of the phone is not the time to discover your recovery is unusable.

Your last alternative is to look at a full backup package (maybe Amanda for Linux based things, Windows has plenty of options) that will recover your system from bare metal. A bare-metal recovery usually allow you to boot from a recovery disk and allow you to pull data up from your most recent backups. Ghost and other imaging tools are really imaging utilities, not really backups of your data (plus the images take larger amounts of data and like I said, can have trouble when disks change brand or size in the process of needing to recover). Commercial backups allow differential backups and a system state backup to help with recovery.

The thing I've come to prefer now is to find a way to separate as much as possible the OS from applications from data. Then you can get the OS running separately from your data, and make it easier to recover or move to a new system (this is kind of what Devil Linux does in that the OS runs in memory from a CD, while data runs from a drive or external drive, so your data can be migrated or backed up easily while the OS is just contained on a CD. Linux can be set up to have the OS set up for /root, /bin, /sbin, (system binaries) and another disk to hold data and logs.

Hope some of this might be useful to you...

Answer 2

You could do something like the hardware manufacturers do, create a recovery partition you can boot from when the primary partition becomes damaged. This would allow you to boot from something known, and have the necessary tools there to restore the main partition to a known good place, either a Ghost image, or something similar.

If you do not have access to the host at a low enough level to do this, there are also KVM switches out there that work over TCP/IP. They are pricey, but cheaper than sending a tech out every time something goes wrong. This might be similar to the AMT you mention, but I have no experience with AMT.

Answer 3

From many years ago my method for refreshing large numbers of machines was to create a bootable CD (yeah, images where smaller back then) containing Ghost and the standard machine image. A batch script would launch Ghost from the command line with the options to copy the image and exit on completion. After that the batch job would eject the CD and reboot. Technically there was a bootup menu at the beginning that gave the user 30 seconds to choose to boot normally, before it automatically started the imaging. While it was used for quick refreshing (make a few copies and just go around popping them into machines and let them do their thing) the lack of any need for user interaction, right up to the system rebooting and running on the image, would seem to put in into the area you are looking.