2
mv /lib64/libkeyutils.so.1.9 /root
service sshd restart
Stopping sshd:                                             [  OK  ]
Starting sshd: /usr/sbin/sshd: error while loading shared libraries: libkeyutils.so.1: cannot open shared object file: No such file or directory
                                                           [FAILED]

How do I remove it from SSHD?

Need to fix this: http://www.webhostingtalk.com/showpost.php?p=8548338&postcount=4

REF about this exploit now that I've heard about it: http://blog.solidshellsecurity.com/2013/02/18/0day-linuxcentos-sshd-spam-exploit-libkeyutils-so-1-9/

They are not logging in with root, nor are they even spawning a bash process. If the lib is moved out, and sshd is restarted they cannot login anymore fwiw.

The key is finding out how they are getting in. Fully upgraded, ssh key restricted sshd, on non-standard ports are being compromised. None of my customers are, but I have been getting alot of sales inquiries with this issue so I don't know the full history of the machines.

[/lib64]# rpm -vV openssh
.........    /etc/ssh
.........  c /etc/ssh/moduli
.........    /usr/bin/ssh-keygen
.........    /usr/libexec/openssh
.........    /usr/libexec/openssh/ssh-keysign
.........    /usr/share/doc/openssh-5.3p1
.........  d /usr/share/doc/openssh-5.3p1/CREDITS
.........  d /usr/share/doc/openssh-5.3p1/ChangeLog
.........  d /usr/share/doc/openssh-5.3p1/INSTALL
.........  d /usr/share/doc/openssh-5.3p1/LICENCE
.........  d /usr/share/doc/openssh-5.3p1/OVERVIEW
.........  d /usr/share/doc/openssh-5.3p1/PROTOCOL
.........  d /usr/share/doc/openssh-5.3p1/PROTOCOL.agent
.........  d /usr/share/doc/openssh-5.3p1/README
.........  d /usr/share/doc/openssh-5.3p1/README.dns
.........  d /usr/share/doc/openssh-5.3p1/README.nss
.........  d /usr/share/doc/openssh-5.3p1/README.platform
.........  d /usr/share/doc/openssh-5.3p1/README.privsep
.........  d /usr/share/doc/openssh-5.3p1/README.smartcard
.........  d /usr/share/doc/openssh-5.3p1/README.tun
.........  d /usr/share/doc/openssh-5.3p1/TODO
.........  d /usr/share/doc/openssh-5.3p1/WARNING.RNG
.........  d /usr/share/man/man1/ssh-keygen.1.gz
.........  d /usr/share/man/man8/ssh-keysign.8.gz
[/lib64]# rpm -vV openssh-clients
S.5....T.  c /etc/ssh/ssh_config
.........    /usr/bin/.ssh.hmac
.........    /usr/bin/scp
.........    /usr/bin/sftp
.........    /usr/bin/slogin
.........    /usr/bin/ssh
.........    /usr/bin/ssh-add
.........    /usr/bin/ssh-agent
.........    /usr/bin/ssh-copy-id
.........    /usr/bin/ssh-keyscan
.........  d /usr/share/man/man1/scp.1.gz
.........  d /usr/share/man/man1/sftp.1.gz
.........  d /usr/share/man/man1/slogin.1.gz
.........  d /usr/share/man/man1/ssh-add.1.gz
.........  d /usr/share/man/man1/ssh-agent.1.gz
.........  d /usr/share/man/man1/ssh-copy-id.1.gz
.........  d /usr/share/man/man1/ssh-keyscan.1.gz
.........  d /usr/share/man/man1/ssh.1.gz
.........  d /usr/share/man/man5/ssh_config.5.gz
[/lib64]# rpm -vV openssh-server
.......T.  c /etc/pam.d/ssh-keycat
S.5....T.  c /etc/pam.d/sshd
.........    /etc/rc.d/init.d/sshd
S.5....T.  c /etc/ssh/sshd_config
.........  c /etc/sysconfig/sshd
.........    /usr/libexec/openssh/sftp-server
.........    /usr/libexec/openssh/ssh-keycat
.........    /usr/sbin/.sshd.hmac
.........    /usr/sbin/sshd
.........    /usr/share/doc/openssh-server-5.3p1
.........  d /usr/share/doc/openssh-server-5.3p1/HOWTO.ssh-keycat
.........  d /usr/share/man/man5/moduli.5.gz
.........  d /usr/share/man/man5/sshd_config.5.gz
.........  d /usr/share/man/man8/sftp-server.8.gz
.........  d /usr/share/man/man8/sshd.8.gz
.........    /var/empty/sshd

and

[/lib64]# rpm -qf /lib64/libkeyutils.so.1.9
file /lib64/libkeyutils.so.1.9 is not owned by any package
[/lib64]# rpm -vV keyutils-libs
....L....    /lib64/libkeyutils.so.1
.........    /lib64/libkeyutils.so.1.3
.........    /usr/share/doc/keyutils-libs-1.4
.........  d /usr/share/doc/keyutils-libs-1.4/LICENCE.LGPL
Tiffany Walker
  • 6,681
  • 14
  • 56
  • 82

1 Answers1

3

Your SSH daemon and system may be compromised!!

You can't trust the existing SSH daemon installed on the server.

For a quick check, run an RPM verification of the existing package. You can do this with:

rpm -vV openssh-server
rpm -vV openssh-clients
rpm -vV openssh

Grep the output of each of those commands for S\.5. This will tell you if the binaries have changed.

A temp fix is to reinstall your openssh setup, but that's beyond the scope of this question. See the following...

How do I deal with a compromised server?

sshd immediately core dumps after fsck repair

Entries in `/etc/inittab` below last line - possible hack?

Community
  • 1
ewwhite
  • 197,159
  • 92
  • 443
  • 809
  • I ran a MD5 on the sshd binary and it matches a clean server. posted output above all looks find – Tiffany Walker Feb 08 '13 at 18:06
  • `rpm -qf /lib64/libkeyutils.so.1.9` and `rpm -vV keyutils-libs` – ewwhite Feb 08 '13 at 18:08
  • posted above... – Tiffany Walker Feb 08 '13 at 18:11
  • After looking at the output, it looks like 1.3 is the real version and 1.9 is not really suppose to be there? Do I RPM remove it? – Tiffany Walker Feb 08 '13 at 18:19
  • `rm` the file and reinstall openssh. – ewwhite Feb 08 '13 at 18:31
  • How do I uninstall openssh and reinstall? Via YUM? – Tiffany Walker Feb 08 '13 at 19:31
  • @TiffanyWalker [See this link.](http://serverfault.com/questions/382093/sshd-immediately-core-dumps-after-fsck-repair/382095#382095) – ewwhite Feb 08 '13 at 19:39
  • All I got was "Installed package openssh-5.3p1-81.el6_3.x86_64 (from updates) not available." so guess this means need to manually do it in some way? – Tiffany Walker Feb 08 '13 at 19:43
  • @TiffanyWalker, that could mean simply that that version has been replaced by a newer one. Try just updating. – vonbrand Feb 19 '13 at 00:01
  • In any case, whatever forensic analysis is to be done should be made from a clean start, e.g. use installation media for rescue and be careful not to run anything on the suspect disk. In any case, the only course of action that would let me sleep at night would be a full reinstall. My condolences. – vonbrand Feb 19 '13 at 00:10
  • @ewwhite: bad advice to rm the file, as there's typically a symlink pointing to it hence applications will fail (ssh, sshd, nslookup, slogin, svn, etc) - see the WebHostingTalk thread for correct way to do this, which requires re-installing the relevant SSH packages, ensuring libkeyutils 1.3 is correct and has symlink, then renaming the file, then ensuring ssh can be run correctly, and then doing a reboot. There are some people who just rm'ed the file and now can't get in via SSH on reboot, so be sure there's another way in (hosting control panel, webmin, etc). – RichVel Feb 19 '13 at 13:12
  • I don't agree this is an exact duplicate of the compromised server canonical question - it's an ongoing issue with many people trying to find out exactly how the attackers are getting in. Specific remediation steps, and analysis of how the attack is accomplished, are much more useful than generic compromised server advice. – RichVel Feb 19 '13 at 13:14