1

So I got hit by a script kitte... Fortunately the box is Ubuntu and was able to replace w/ binaries from a comparable system, however,

Some of the files I couldn't delete, and am still stumped on this. The hijacked files are sitting in the /_bin directory which is writeable by root.

nathan@db-0:~$ ls -ld !$
ls -ld /_bin
drwxr-xr-x 2 root root 4096 Mar 12 18:00 /_bin

Ok, those are the perms on the directory, now for the files within:

nathan@db-0:~$ ls -l /_bin
total 268
-rwxr-xr-x 1 root root  39696 Nov 19 22:25 ls
-rwxr-xr-x 1 root root 119800 Mar 31  2012 netstat
-rwxr-xr-x 1 root root 101240 Dec 12  2011 ps

Now when I try to delete one of these files (as root):

root@db-0:/home/nathan# rm /_bin/ls
rm: cannot remove `/_bin/ls': Operation not permitted

Or if I try to delete the entire _bin directory (again as root):

root@db-0:/home/nathan# rm -rf /_bin
rm: cannot remove `/_bin/ls': Operation not permitted
rm: cannot remove `/_bin/netstat': Operation not permitted
rm: cannot remove `/_bin/ps': Operation not permitted

So how can I delete these files?

Edit:

Sure enough the immutable bit has been set, however, removing it does not let me delete the files.

root@db-0:/home/nathan# lsattr /_bin
s---ia--------- /_bin/ls
s---ia--------- /_bin/netstat
s---ia--------- /_bin/ps

root@db-0:/home/nathan# chattr -R -i /_bin
root@db-0:/home/nathan# lsattr /_bin
s----a--------- /_bin/ls
s----a--------- /_bin/netstat
s----a--------- /_bin/ps

root@db-0:/home/nathan# rm -rf /_bin
rm: cannot remove `/_bin/ls': Operation not permitted
rm: cannot remove `/_bin/netstat': Operation not permitted
rm: cannot remove `/_bin/ps': Operation not permitted

Also verified /_bin doesn't have immutable bit:

root@db-0:/home/nathan# lsattr -d /_bin
--------------- /_bin
Drifter104
  • 3,773
  • 2
  • 25
  • 39
quickshiftin
  • 2,125
  • 5
  • 27
  • 41
  • 7
    Reinstall the system from scratch, don't try to repair it. You'll be never sure to have eliminated everything. – Sven Mar 12 '13 at 18:16
  • Can you do `strace -v rm /_bin/ls` and as it's longer output, put it on the web like pastebin.com? – ott-- Mar 12 '13 at 18:34
  • 2
    I say we take off and nuke the entire site from orbit. It's the only way to be sure. http://www.youtube.com/watch?v=aCbfMkh940Q – Deer Hunter Mar 12 '13 at 19:05
  • The answer you accepted says there are other extended attributes that might prevent removal... there might also be `/_bin/.*` files with funky attributes. – vonbrand Mar 12 '13 at 22:14
  • Good man @vonbrand, though that wasn't the case here an old colleague once told me of a case where there was a `...` directory - tricky kitty! – quickshiftin Mar 13 '13 at 00:06
  • @quickshiftin, we once had a `.. ` (dot dot space) directory full of malware in `/bin`. – vonbrand Mar 13 '13 at 00:11

2 Answers2

10

Most likely the attacker has set the immutable attribute on the files and directory. This is commonly done by rootkits to make cleanup more difficult.

To confirm this, try:

lsattr /_bin

To remove the immutable attribute, use:

chattr -R -i /_bin

You'll also want to clear the a and s attributes, since these may affect your ability to remove the files.

chattr -R -i -a -s /_bin

See the chattr man page for a full explanation of what all the attributes are and what they do.

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
-1

Looks like the sticky bit is still there.

chmod -t /_bin
AWippler
  • 1,065
  • 1
  • 12
  • 32