12

I was SSHed into a remote box as root when I ran the following command:

ln -sf /nonexistent /.../libc.so

Immediately my prompt started throwing errors:

basename: could not find shared library

I can't even run anything:

root@toastbox# ls
ls: could not find shared library

How can I fix this? I have two SSH sessions open with Bash, but no other processes accessible. I have a cross-compiler for the target on my local machine, but no way to SCP files to the remote end anymore.

EDIT: There are no other copies of libc on this box; I overwrote the real libc file. Some things still work: I can echo, and I can use tab-completion to emulate ls. But normal programs (mv, rm, etc.) are MIA.

nneonneo
  • 171,345
  • 36
  • 312
  • 383
  • 5
    This might be more appropriate for Super User. – Veedrac Dec 15 '14 at 17:35
  • Remove (unlink) the litany of `nonexistent` soft-links you created and reinstall libc. (and hope, if necessary, you have a competent set of hands at the remote site if remote access tanked) – David C. Rankin Dec 15 '14 at 17:37
  • @DavidC.Rankin: I would, but I can't run anything... – nneonneo Dec 15 '14 at 17:37
  • Remote site, find someone competent to stuff a install/recovery disk into the box, then reinstall libc. I think it will be darn near impossible to manually unlink and recover the original setup if your `ln -sf` reassigned existing links. Unless you have some type of snapshot on your filesystem (btrfs, etc..), reinstall of libc (and potentially other packages) looks like the only option. – David C. Rankin Dec 15 '14 at 17:40
  • On my openSuSE box, the only links you would have clobbered would have been `/lib/libc.so.6 -> libc-2.18.so` and `/lib64/libc.so.6 -> libc-2.18.so` if you can back those out (with appropriate version, etc.), you may be able to save the day. On Arch it is `/usr/lib/libc.so.6 -> libc-2.20.so`. Some recovery of this type would be the only feasible way short of reinstall of libc. – David C. Rankin Dec 15 '14 at 17:44
  • 3
    Does `echo -en "data" >> file` work, and do you have a local copy of the clobbered `libc`? If so, a highly unusual recovery I propose is that you write yourself a small tool locally that accepts the unbroken libc and generates a sequence of `echo -en "\xDA\xTA\xBL\x0C\xKS" >> libc.so` commands. You'd then truncate your remote `libc` (`echo -n "" > libc`) and run that sequence of commands, and at the end you'd have a restored `libc`. Highly unusual, true, but requires no presence at the remote site. – Iwillnotexist Idonotexist Dec 15 '14 at 18:00
  • @IwillnotexistIdonotexist: My `libc` is symlinked away; I can't just write to it (and the directory that `nonexistent` is in also does not exist). Nevertheless, this is a promising solution, I am pursuing it. – nneonneo Dec 15 '14 at 18:02
  • No code, no cookie. Migrate to SuperUser.com – leppie Dec 15 '14 at 18:07
  • @nneonneo I almost didn't think I'd be taken seriously! Advice: You may test this command-generating tool locally, but make sure your generated commands don't specify the path of your local `libc` during your local test! – Iwillnotexist Idonotexist Dec 15 '14 at 18:07
  • @IwillnotexistIdonotexist: Yes, I will try not to destroy my local `libc` during this process :) – nneonneo Dec 15 '14 at 18:08
  • @nneonneo As one extra idea, you may try bootstrapping yourself by inserting a statically-linked `netcat`, `mv`, `rm`, `ln` or `busybox` by the same general mechanism. – Iwillnotexist Idonotexist Dec 15 '14 at 18:10
  • 1
    @IwillnotexistIdonotexist: Actually already started on that approach. I have the echo-encoder mostly written now... – nneonneo Dec 15 '14 at 18:11
  • 1
    The remote `echo` only supports octal escapes for some silly reason though, so it makes the code bloat a lot. But this insane approach may just work!! – nneonneo Dec 15 '14 at 18:12
  • @nneonneo Does the remote `echo` support `-n` (no new lines) and `-e` (Interpret escape sequences)? – Iwillnotexist Idonotexist Dec 15 '14 at 18:13
  • @IwillnotexistIdonotexist: Yes, thankfully it does. I have already tested that. It also should support `>>`. (I can't actually check, but I'm going to hope it does!) – nneonneo Dec 15 '14 at 18:14
  • 1
    @IwillnotexistIdonotexist Hahaha! It works! I just wrote and copied over an `mv` that runs. Now I am copying over `libc`...it's going to take some time because pasting in 10000 `echo` commands takes ages. But this looks like it will work!! – nneonneo Dec 15 '14 at 18:25
  • 1
    For the future, keeping a statically linked busybox around is very handy for making this kind of situation easier to recover from. (And I echo the argument that StackOverflow is the wrong place for this discussion -- it's a system administration question, not a programming one). – Charles Duffy Dec 15 '14 at 18:43

1 Answers1

7

I discovered that I could still write to files by using echo and redirection (thanks Iwillnotexist Idonotexist!). Further, echo -ne lets me write arbitrary bytes to a file. I can therefore truncate a file with echo -ne '' > file, then repeatedly write to it with

echo -ne '\001' >> /file

Using this approach, I can overwrite any executable present on the system (since I'm still root) in this way.

I compiled a simple program to rename a file:

#include <unistd.h>
int main(int argc, char **argv) { return rename(argv[1], argv[2]); }

using cross-gcc -static mv.c mv (eliminating the libc.so dependency). Then, I wrote a script to encode any binary file as a series of echo commands (limited by the length that readline will allow me to enter):

# Encode a file as a series of echo statements.

# settings
maxlen = 1020
infile = '/tmp/mv'
outfile = '/usr/bin/mv'

print "echo -ne '' > %s" % outfile

template = "echo -ne '%%s' >> %s" % outfile
maxchunk = maxlen - len(template % '')
pos = 0
data = open(infile, 'rb').read()

transtable = {}
for i in xrange(256):
    c = chr(i)
    if i == 0:
        transtable[c] = r'\0'
    elif c.isalpha():
        transtable[c] = c
    else:
        transtable[c] = r'\0%o' % i

while pos < len(data):
    chunk = []
    chunklen = 0
    while pos < len(data):
        bit = transtable[data[pos]]
        if chunklen + len(bit) < maxchunk:
            chunk.append(bit)
            chunklen += len(bit)
            pos += 1
        else:
            break
    print template % ''.join(chunk)

I used my echo encoder to generate a series of echo commands which I mass-pasted into the ssh session. These look like

echo -ne '' > /usr/bin/mv
echo -ne '\0177ELF\01\01\01\0\0\0\0\0\0\0\0\0\02\0\050\0\01\0\0\0\0360\0200\0\0\064\0\0\0\030Q\05\0\0\0\0\05\064\0\040\0\05\0\050\0\034\0\033\0\01\0\0\0\0\0\0\0\0\0200\0\0\0\0200\0\0P\03\01\0P\03\01\0\05\0\0\0\0\020\0\0\01\0\0\0\0\017\01\0\0\0237\01\0\0\0237\01\0x\02\0\0X\046\0\0\06\0\0\0\0\020\0\0Q\0345td\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\06\0\0\0\0\0\0\0\01\0\0p\0244\0356\0\0\0244n\01\0\0244n\01\0\0350\010\0\0\0350\010\0\0\04\0\0\0\04\0\0\0R\0345td\0\017\01\0\0\0237\01\0\0\0237\01\0\0\01\0\0\0\01\0\0\06\0\0\0\040\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\020\0265\04\034\0\040\0\0360\053\0371\040\034\016\0360r\0375\0134\0300\0237\0345\0H\055\0351X\060\0237\0345\04\0260\0215\0342\020\0320M\0342\014\0300\0217\0340\03\060\0234\0347\024\060\013\0345D\060\0237\0345\04\0\0213\0342\03\060\0234\0347\020\060\013\0345\070\060\0237\0345\0\020\0240\0343\03\060\0234\0347\014\060\013\0345\054\060\0237\0345\03\060\0234\0347\010\060\013\0345\044\060\0237\0345\03\040\0234\0347\024\060K\0342\0223\072\0\0353\04' >> /usr/bin/mv
echo -ne '\0320K\0342\0\0210\0275\0350\0350\036\01\0\0174\0377\0377\0377\0200\0377\0377\0377\0204\0377\0377\0377\0210\0377\0377\0377\0214\0377\0377\0377\0H\055\0351\04\0260\0215\0342\010\0320M\0342\010\0\013\0345\014\020\013\0345\014\060\033\0345\04\060\0203\0342\0\040\0223\0345\014\060\033\0345\010\060\0203\0342\0\060\0223\0345\02\0\0240\0341\03\020\0240\0341\06\0\0\0353\0\060\0240\0341\03\0\0240\0341\04\0320K\0342\0\0210\0275\0350\0\0\0\0\0\0\0\0\0\0\0\0\0220\0\055\0351\046p\0240\0343\0\0\0\0357\0220\0\0275\0350\0\0\0260\0341\036\0377\057Qr\072\0\0352\0\0\0240\0341\020\0265\04\034\0\0360\014\0370\04\0140\01\040\0100B\020\0275\020\0265\03\034\0377\063\02\0333\0100B\0377\0367\0361\0377\020\0275\020\0265\02K\0230G\010\060\020\0275\0300F\0340\017\0377\0377\0360\0265\031N\0203\0260\034\034\0176D\07\034\01\0222\0\0360\0253\0371\045h\0\0340\0230G\04\065\053h\0\053\0372\0321\0345h\0\0340\0230G\04\065\053h\0\053\0372\0321eh\0\0340\0230G\04\065\053h\0\053\0372\0321\075\034\0200\0315y\034\0210\0' >> /usr/bin/mv
...

I tested the replacement mv a few times to make sure it worked (using Bash tab-completion as a substitute for ls), and then used the echo encoder to write a replacement libc.so to a temporary directory. Finally, I moved the replacement libc.so into the right place using the static mv I pushed.

And success! It might've taken about an hour, but my box is back up and running, with no casualties save for one clobbered /usr/bin/mv :)

Community
  • 1
  • 1
nneonneo
  • 171,345
  • 36
  • 312
  • 383
  • Congratulations on implementing this strategy successfully! – Iwillnotexist Idonotexist Dec 15 '14 at 18:42
  • 2
    @IwillnotexistIdonotexist: Yeah, I'm super happy about this. Thank you so much for your help! – nneonneo Dec 15 '14 at 18:43
  • Man you killed three birds with one stone on this! ([on-the-road](http://winterbash2014.stackexchange.com/on-the-road), [business-in-the-front-party-in-the-back](http://winterbash2014.stackexchange.com/business-in-the-front-party-in-the-back), [selfie](http://winterbash2014.stackexchange.com/selfie))! – Scott Solmer Dec 18 '14 at 01:05
  • I wonder how you were able to run the python script if `libc.so` was corrupted/missing? – Maitre Bart Feb 24 '21 at 17:00
  • I was `ssh`ed into the machine (as I mentioned in the question) and luckily did not quit the session at any point. I ran Python on my own machine and pasted the commands into the terminal running `ssh`. – nneonneo Feb 24 '21 at 18:07