/usr/bin/sshd isn't linked against PAM on one of my systems. What is wrong and how can I fix it?

Question

I'm using AD as my user account server with ldap.

Most of the servers run with UsePam yes except this one,

it has lack of pam support on sshd.

root@linserv9:~# ldd /usr/sbin/sshd 
    linux-vdso.so.1 =>  (0x00007fff621fe000)
    libutil.so.1 => /lib/libutil.so.1 (0x00007fd759d0b000)
    libz.so.1 => /usr/lib/libz.so.1 (0x00007fd759af4000)
    libnsl.so.1 => /lib/libnsl.so.1 (0x00007fd7598db000)
    libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007fd75955b000)
    libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007fd759323000)
    libc.so.6 => /lib/libc.so.6 (0x00007fd758fc1000)
    libdl.so.2 => /lib/libdl.so.2 (0x00007fd758dbd000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fd759f0e000)

I have this packages installed

root@linserv9:~# dpkg -l|grep -E 'pam|ssh'
ii  denyhosts                             2.6-2.1                      an utility to help sys admins thwart ssh hac
ii  libpam-modules                        0.99.7.1-5ubuntu6.1          Pluggable Authentication Modules for PAM
ii  libpam-runtime                        0.99.7.1-5ubuntu6.1          Runtime support for the PAM library
ii  libpam-ssh                            1.91.0-9.2                   enable SSO behavior for ssh and pam
ii  libpam0g                              0.99.7.1-5ubuntu6.1          Pluggable Authentication Modules library
ii  libpam0g-dev                          0.99.7.1-5ubuntu6.1          Development files for PAM
ii  openssh-blacklist                     0.1-1ubuntu0.8.04.1          list of blacklisted OpenSSH RSA and DSA keys
ii  openssh-client                        1:4.7p1-8ubuntu1.2           secure shell client, an rlogin/rsh/rcp repla
ii  openssh-server                        1:4.7p1-8ubuntu1.2           secure shell server, an rshd replacement
ii  quest-openssh                         5.2p1_q13-1                  Secure shell
root@linserv9:~# 

What I'm doing wrong?

thanks.

Edit:

root@linserv9:~# cat  /etc/pam.d/sshd 
# PAM configuration for the Secure Shell service

# Read environment variables from /etc/environment and
# /etc/security/pam_env.conf.
auth       required     pam_env.so # [1]
# In Debian 4.0 (etch), locale-related environment variables were moved to
# /etc/default/locale, so read that as well.
auth       required     pam_env.so envfile=/etc/default/locale

# Standard Un*x authentication.
@include common-auth

# Disallow non-root logins when /etc/nologin exists.
account    required     pam_nologin.so

# Uncomment and edit /etc/security/access.conf if you need to set complex
# access limits that are hard to express in sshd_config.
# account  required     pam_access.so

# Standard Un*x authorization.
@include common-account

# Standard Un*x session setup and teardown.
@include common-session

# Print the message of the day upon successful login.
session    optional     pam_motd.so # [1]

# Print the status of the user's mailbox upon successful login.
session    optional     pam_mail.so standard noenv # [1]

# Set up user limits from /etc/security/limits.conf.
session    required     pam_limits.so

# Set up SELinux capabilities (need modified pam)
# session  required     pam_selinux.so multiple

# Standard Un*x password updating.
@include common-password

Edit2: UsePAM yes fails

With this configuration ssh fails to start :

root@linserv9:/home/admmarc# cat /etc/ssh/sshd_config |grep -vE "^[ \t]*$|^#"
Port 22
Protocol 2
ListenAddress 0.0.0.0
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys
ChallengeResponseAuthentication yes
UsePAM yes
Subsystem       sftp    /usr/lib/sftp-server
root@linserv9:/home/admmarc# 

The error it gives is as follows

root@linserv9:/home/admmarc# /etc/init.d/ssh start
 * Starting OpenBSD Secure Shell server sshd
/etc/ssh/sshd_config: line 75: Bad configuration option: UsePAM
/etc/ssh/sshd_config: terminating, 1 bad configuration options
   ...fail!
root@linserv9:/home/admmarc#

Answer 1

It looks like your /usr/sbin/sshd binary has been overwritten.

This could mean you have had a security breach, or maybe someone just compiled a version locally and the Ubuntu version was overwritten.

The 1:4.7p1-8ubuntu1.2 (x86) version of the openssh-server definitely is definitely linked against libpam. It is conceivable that the pam support was left out of the 64bit version, but that seems unlikely.

I'd try reinstalling openssh-server:

# First back up the current binary
$ sudo cp /usr/sbin/sshd /root/sshd.bak

# reinstall the Ubuntu version
$ sudo apt-get install --reinstall openssh-server

# compare the two versions
$ sudo sha1sum /usr/sbin/sshd /root/sshd.bak

# In my case, they match:
# 8a3ccd5242380674bc45b887286faa3abb51acdb  /usr/sbin/sshd
# 8a3ccd5242380674bc45b887286faa3abb51acdb  /root/sshd.bak

If yours don't match each other (not mine), there is definitely something fishy going on and you really need to figure out where that version of sshd came from.

If they do match, then I am probably wrong and it is a bug in that version of openssh-server for 64-bit Ubuntu.

Answer 2

Based on the information provided in the question, it looks like your sshd overwriting may not be due to a system compromise, but due to something more benign like a sysadmin trying to change the standard ssh. The reason I suspect this is some admin action, is that the system also has this non-standard ssh package installed:

quest-openssh.

You may run:

dpkg-query -L quest-openssh | grep sshd

and see whether this package contains a file /usr/bin/sshd which has overwritten your openssh server (according to Thedward, and also by my verification, quest openssh installs under /opt, but there's a (low) chance that it was configured differently). In any case, you should ask your sysadmin if he was trying to replace sshd, by what, how, and why.

To sum up my thinking: yes, your sshd has definitely been overwritten, but I think it is a bit hasty to conclude that your system was definitely compromised.

Lastly: how to fix it? Remove the quest-openssh package, reinstall the standard openssh, and ldap-auth-client which uses libpam-ldap support for the standard openssh. Generally, on a debian/ubuntu based system, there's rarely a need to install non-standard, or non-free commercial packages to get open/standard functionality. You can find more information on how to set-up openssh PAM for LDAP here: http://wiki.linuxquestions.org/wiki/Pam_ldap

Answer 3

Does /etc/pam.d/sshd exist? It's part of openssh-server.

Answer 4

If your system was really compromised, a system reinstall might be in order!