10

Alright if anyone wants to play god and work miracles i'm down.

So, i was given the task of making a script that found files that are older than 6 months, zipped them and then deleted them. On my way in making tihs script i ran this : 

find / -type f -mtime -400 ! -mtime -180 | xargs gzip blablabla

And that gave EVERY SINGLE FILE a .gz extension. Now i undid it as soon as i noticed but it was kinda too late. Upon completion of the command none of my bash commands would work because the $PATH variable emptied itself. I tried lots of things before realising what was the problem.

So ater unzipping everything i'm still unable to boot. I've managed to make it to grub rescue, after wich i followed online instructions for :

root (hd0,0)
setup (hd0)
kernel (hd0,0)/boot/vml[...]
initrd (hd0,0)/boot/initrd.im[...]

After wich my linux partially boots but gives me the following errors :

Begin : Running /scripts/init-bottom ... mount : mounting /dev on /root/dev failed : No such file or directory
mount: mounting /sys/ on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed : No such file or directory
Target filesystem doesn't have requrested /sbin/init.
No init found. Try passing init= bootarg.

I've tried repairing the filesystem, i've booted from 3 different LiveCDs/ Rescue disks, i ran boot-repair from 2 different dicsc. I did force fscks...

I'm really out of ideas and i NEED to get this server to at least boot so i can recover my SQL databases. I'm desperate for help, i'll even pay if need be.

I've been lurking forums for 3 days straight all day to find a possible solution and i'm still at the same point... Help please?

Dexirian
  • 430
  • 2
  • 11
  • 3
    if they're mysql db's you don't necessarily need to boot; in that case I'd try to mount the drive as a slave and copy over the /var/lib/mysql directory – user16081-JoeT Oct 15 '13 at 19:54
  • 8
    Clean install on new storage devices. Mount the old drive, transfer data as required. I would bet that repairing will not be worth the effort. – Zoredache Oct 15 '13 at 19:56
  • We tried copying the content of /var/lib/mysql about 2 hours ago, we are having several corrupt databases problems and whatnot because of version differences, i did mount the old drive to transfert everything but we have corruption issues, hence me posting here – Dexirian Oct 15 '13 at 19:59
  • @Dexirian You could ask a question relating the version differences. Seems like a totally independent question to me. (Assuming you did research it already) – Jeroen Oct 15 '13 at 20:04
  • I'm not looking to fix the SQL corruptions issues, im looking to boot on my server. All i want to know is is it possible? If yes how? If not i will another solution – Dexirian Oct 15 '13 at 20:09
  • 7
    This is the point where you restore from backup. And remember not to run around doing non-privileged actions as the root user next time. – Magellan Oct 15 '13 at 20:18
  • This is also the point where you learn to execute your `find` first (without piping it to anything else) to make sure it's only `find`ing what you want it to. Fortunately your lesson doesn't have to be ***too*** painful: [Bob Barker](http://serverfault.com/a/546256/32986) gave you the solution in a nutshell. – voretaq7 Oct 15 '13 at 20:21
  • 1
    `because of version differences,` re-install with the exact same version. `we have corruption issues,` your data may be corrupt. Repairing the system so it is bootable, will not help you if the data has been trashed. If your gzip command compressed your database files while the database was being used corrupt seems inevitable. – Zoredache Oct 15 '13 at 20:34
  • 5
    If your DB software was running while you ran those commands, you might not be able to recover the DBs. Gzip would have happily zipped up the file, then it unlinked it. But your DB software still had the file open and it was committing the changes to it. As soon as it stopped, the file was then removed. – toppledwagon Oct 15 '13 at 20:34
  • 1
    If the GZIP got to the /dev folder (looks like it from you log-messages) or the DB files you better have a good backup. Because that will be your only recovery point. – Tonny Oct 15 '13 at 21:24

3 Answers3

9

The first thing I'd try is running a LiveCD environment and just attempt to ungzip everything, hoping that would return the system to a bootable state. Note: I'd be concerned about potential data corruption if the original gzip process was interrupted.

Otherwise I'd try to migrate the DB to a new system as others have suggested but as you've encountered there may be labor intensive dependency and configuration issues that will need to be resolved individually.

Bob Barker
  • 129
  • 1
  • Quick question : We are unsure of the old SQL database server, and the newer server is using a different Linux Distro. The newer server is running CentOS with WHM, and the old server was either Debian / Unbuntu. So my question is, how can i effectively migrate my SQL databases without corruption and whatnot? – Dexirian Oct 16 '13 at 14:18
8

This will depend on whether the filesystems are repaired enough for you to be able to mount those partitions from a LiveCD. Don't bother trying to boot the system yet. First, mount the partitions and unzip all the .gz files. This will give you working copies of init and system binaries. Then you can use grub to repair the boot sector. Then boot to single user mode and fsck the filesystem again. If that works, you'll have a running system. You'll also have a bunch of unzipped files (like man pages) that really should be zipped, but it's better than having an unbootable system.

If you can't mount the partitions from a LiveCD, then you are unfortunately out of luck. Nothing will recover your system at that point.

Michael Martinez
  • 2,645
  • 3
  • 24
  • 35
6

The general consensus here, that you should just mount the disk in a working system and rescue your files, is not wrong. It's the sensible thing to do. But the other way is more fun, and very educational. I've learned a lot while fighting my way out of messy situations where other people would have just given up and reinstalled from scratch. (Not on a server that other people are depending on though...)

Anyway, so far you have an initramfs(initrd) that runs. That's a good start. But it can't complete the handoff to init because init is now init.gz maybe? To make any progress, it would help to know exactly what Linux distribution you have, so we can look up what tools are available in its initramfs for emergency use.

The error messages you presented look like they could have come from Debian's initramfs. If it is Debian, then you should have gotten an (initramfs) shell prompt on the next line after the last error. If you did, you should look into what's going on with those failed mounts. is /root/dev missing? (/root is where your normal root fs should be mounted during the running of the initramfs)

If you didn't get the shell prompt, then what came after No init found. Try passing init= bootarg. will be interesting. Even if it was nothing but a blinking cursor, that's a clue. If it seems totally frozen, try to get some information on what processes are still around using magic sysrq or Ctrl+ScrollLock.

The Debian initramfs also lets you request a shell at a few special landmarks by adding a break= parameter to the kernel command line. For example, to get a shell before the Running /scripts/init-bottom line, use break=bottom.

Aside: I don't know how the find command could have gzipped every file... it looks correct to me for the purpose of selecting files between 180 and 400 days old.

  • When i do an ls under /root there is nothing found. So i can take it the fs mounting isn't okay on boot? Where can i change it? – Dexirian Oct 16 '13 at 12:59
  • 1
    @Dexirian so you did get a shell prompt (did you have to use `break=bottom`?)... yes, by the time it's trying to mount `/root/dev` and `/root/proc` and `/root/sys`, `/root` should be the real root filesystem. There must have been an error message earlier, about failing to mount it. Did you include a `root=` parameter in the kernel command line? My memory is a little fuzzy on this point but I think the `root (hd0,0)` just tells grub where to find its support files, and you still need to tell the kernel separately where the root is. –  Oct 16 '13 at 13:10
  • Yes i did use root=, kernel= initrd=, and setup=, i did not have to use break=bottom. And i did not spot an earlier failed mount message, as it scrolls really quickly – Dexirian Oct 16 '13 at 14:15
  • @Dexirian Is console scrollback available? Shift+PgUp. And can you mount it fromt he `(initramfs) ` prompt, something like `mount -r /dev/sda1 /root`? `cat /proc/partitions` to see what disks are available. –  Oct 16 '13 at 14:34
  • I was actually able to boot re-doing the whole process over, thanks a lot everyone for all your help! This website just saved me so much time and money, there's no words to express my thanks. – Dexirian Oct 16 '13 at 15:03
  • 1
    Welcome to the world of system administration. – Michael Martinez Oct 16 '13 at 17:32
  • I like thus far ;) – Dexirian Oct 16 '13 at 18:12
  • Not a bad way to make a living ;-) – Michael Martinez Oct 16 '13 at 19:13