Through a series of unfortunate events, a badly-configured script was executed with elevated privileges, which then executed the following command:
/usr/bin/rsync --recursive --links --copy-unsafe-links --delete --perms --times --group --owner --bwlimit=20000 / /
Clearly, there were supposed to be some paths at the end there, but it ran targetting the root of the file system.
I'm now unable to ssh into the server, and the (virtual) console is inoperative (just black screen with flashing cursor). I do have a terminal session still open, and I also have an scp session which is able to traverse the file structure, so I'm busy backing up as much important stuff as I can.
However, I'd really like to see if it's possible to rescue it, as it does appear to still be running—it has 3 virtual kvm guests that are still running OK.
The only commands which I seem to be able to use are cd and echo, so for example:
[root ~] # cd /bin
[root bin] # ls
-bash: /bin/ls: No such file or directory
[root bin] # echo *
bash btrfs btrfsck btrfs-debug-tree btrfs-find-root btrfs-image btrfs-map-logical btrfs-select-super btrfstune btrfs-zero-log bunzip2 busybox bzcat bzcmp bzdiff bzegrep bzexe bzfgrep bzgrep bzip2 bzip2recover bzless bzmore cat chacl chgrp chmod chown chvt cp cpio dash date dd df dir dmesg dnsdomainname domainname dumpkeys echo ed efibootdump efibootmgr egrep false fgconsole fgrep findmnt fsck.btrfs fuser fusermount getfacl grep gunzip gzexe gzip hostname ip journalctl kbd_mode keyctl kill kmod less lessecho lessfile lesskey lesspipe ln loadkeys login loginctl lowntfs-3g ls lsblk lsmod mkdir mkfs.btrfs mknod mktemp more mount mountpoint mt-gnu mv nano nc.openbsd netstat networkctl nisdomainname ntfs-3g ntfs-3g.probe ntfscat ntfscluster ntfscmp ntfsfallocate ntfsfix ntfsinfo ntfsls ntfsmove ntfsrecover ntfssecaudit ntfstruncate ntfsusermap ntfswipe open openvt ping ping4 ping6 plymouth ps pwd rbash readlink red rm rmdir rnano run-parts sed setfacl setfont setupcon sh sh.distrib sleep ss static-sh stty su sync systemctl systemd-ask-password systemd-escape systemd-hwdb systemd-inhibit systemd-machine-id-setup systemd-notify systemd-sysusers systemd-tmpfiles systemd-tty-ask-password-agent tar tempfile touch true udevadm ulockmgr_server umount uname uncompress unicode_start vdir wdctl which whiptail ypdomainname zcat zcmp zdiff zegrep zfgrep zforce zgrep zless zmore znew
[root bin] #
As far as I can tell, all the files which should be there are there (in /bin, /boot, /sbin, etc., just by comparing the output of echo * with another working system). Tab-filename completion is also working, but I cannot execute anything. I'm assuming the rsync has messed up file mode / attributes / something else (I can't even tell if the script is still running, as ps is also unavailable).
I'm also assuming if I reboot it, it will fail to start, but I can hopefully attach a Live CD and start the rescue from there.
My questions are (a) is there something else I can try before rebooting, given that it may never come back up, and (b) is there a good approach to start fixing whatever damage has been done?
