My company runs small NAS boxes that we distribute into businesses and homes for personal or business networking use. These machines run Ubuntu Server 10.04 and are set to boot from a flash drive. The bulk of the storage and shares is on a 4-drive RAID5 array. We clone the flash drives using dd and swap them out for quick and easy repairs to issues with the OS. Some of our clients can't afford to have their box off even for a few minutes for a reboot. So we would like to know how to get started with creating a "hot swappable" flash drive. I was considering running a small OS with just the basic share, ssh, and FTP on it from the RAID5 array, and use a USB signal to run a script instructing the machine to switch the bootloader to this "tiny" OS. So while the machine runs on this make-shift, temporary OS, we switch out the flash drive with the primary OS, and run the USB signal again to switch the bootloader back. So I guess my question would be if there's software made for this already, and if not, what path should I take to get started?
2 Answers
So ... this is actually a neat question.
I'll first tell you why it won't work, then I'll tell you how it might work. :-)
Your "OS" (more specifically your kernel) provides all of the necessary interfaces for your devices. Including your disk arrays. It also keeps a list of descriptors that tell processes where to send their data.
So, when you talk about "hot swapping" your "OS" you are actually asking to replace your kernel (and all of it's associated libraries and configurations) during operations. You CAN have an OS in operation, and another stored on a device. But you cannot hotswap a standard kernel during operation.
BUT, you can look at a new technology called ksplice. Its goal is to offer a reboot-less kernel. I am not familiar with your current implementation, or how stable ksplice currently is, but it should get you started down the right path. And Canonical is even one of their major supporters!
Good luck!
- 9,899
- 4
- 32
- 56
If the problem is with the o/s configuration swapping out the drive does nothing. You cannot alter the state of a running O/S that simply without doing something like at least restarting services.
With regards to replacing a faulting drive with a new one on USB for example that is possible to do.
IF you create the USB disk as a volume group in LVM2 then you can use the pvmove command to achieve this effect.
The process would be:
- Insert equal or greater sized new USB disk.
- Using vgextend add the said disk to the existing volume group.
- Using pvmove switch the extents from the old root partition over to the new USB disk.
- Using vgreduce remove old old USB disk from the volume group.
- Remove old drive.
I will mention again, this does nothing at all to fix any underlying issues caused by the state of the server which is likely where most problems' root cause occurs.
This in addition does nothing to help you fix existing kit that is not LVM enabled. Just stuff you would send out now.
If you want to offer a highly available solution design one in the first place. Build a redundant NAS. You can use drbd to do this. Then you can safely failover to do as much offline work on the problem box as you want.
- 23,357
- 3
- 55
- 72