6

Thanks for taking the time to check out my problem.

I'm currently working on an issue that has only appeared once before. Back on Jan 3rd when this first appeared we were able to reboot the server and everything seemed fine, but now it is back. This is a production database system, so finding a window to reboot can sometimes be difficult. I'm hoping to get a firm grasp on what may actually be happening this time before we reboot again in a few days to provide another temp fix for the issue. Here we go...

User authentication for the system in question is handled with LDAP via Red Hat Directory Server 9. The issue described below is only seen on this one server and even it's counterpart that shares the database doesn't display the same symptoms. As of right now, no LDAP accounts are able to authenticate and log into the server. LDAP auth is being handled by SSSD, which is currently unable to be stopped or restarted. When attempting to do either the SSH console becomess unresponsive. (ctrl-c is unable to exit the issued command)

PS shows the usual sssd related processes are running, but attempting kill -9 on them doesn't seem to do successfully stop any of them.

ps aux | grep sss | grep -v grep
root      1150  0.0  0.0 150828  2908 ?        D    09:05   0:00 /usr/libexec/sssd/sssd_nss -d 0 --debug-to-files
root      7025  0.0  0.0  93616  2504 pts/2    D    16:18   0:00 /usr/sbin/sssd -f -D
root     11148  0.0  0.0 179436  5672 ?        D    Jan08  16:22 /usr/libexec/sssd/sssd_be -d 0 --debug-to-files --domain default
root     32700  0.0  0.0 150784  2908 ?        D    10:10   0:00 /usr/libexec/sssd/sssd_pam -d 0 --debug-to-files

Using strace getent -s sss passwd I can see that some of the connection attempts are being refused, but I'm not really sure what to do about them.

connect(3, {sa_family=AF_FILE, path="/var/lib/sss/pipes/nss"...}, 110) = -1 ECONNREFUSED (Connection refused)
close(3)                                = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 3
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
fcntl(3, F_GETFD)                       = 0
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
connect(3, {sa_family=AF_FILE, path="/var/lib/sss/pipes/nss"...}, 110) = -1 ECONNREFUSED     (Connection refused)
close(3)                                = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 3
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
fcntl(3, F_GETFD)                       = 0
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
connect(3, {sa_family=AF_FILE, path="/var/lib/sss/pipes/nss"...}, 110) = -1 ECONNREFUSED (Connection refused)

Checking lsof | head -n1; lsof | grep /var/lib/sss/pipes/ shows far less open pipes between the good and the bad system. The PIDs for these pipes are the same reported from ps aux, so attempting kill -9 on them has been fruitless as well.

bad sssd

lsof | head -n1; lsof | grep /var/lib/sss/pipes/
COMMAND     PID         USER   FD      TYPE             DEVICE    SIZE/OFF       NODE NAME
sssd_be   11148         root   15u     unix 0xffff8806635911c0         0t0   31817638 /var/lib/sss/pipes/private/sbus-dp_default.11148
sssd_be   11148         root   16u     unix 0xffff880d443d6180         0t0   31783555 /var/lib/sss/pipes/private/sbus-dp_default.11148
sssd_be   11148         root   17u     unix 0xffff880c536d94c0         0t0   31783560 /var/lib/sss/pipes/private/sbus-dp_default.11148

good sssd

lsof | head -n1; lsof | grep /var/lib/sss/pipes/
COMMAND     PID         USER   FD      TYPE             DEVICE    SIZE/OFF       NODE NAME
sssd      26793         root   13u     unix 0xffff88030b5d8c40         0t0 3248762734 /var/lib/sss/pipes/private/sbus-monitor
sssd      26793         root   14u     unix 0xffff8808cc064bc0         0t0 3248762735 /var/lib/sss/pipes/private/sbus-monitor
sssd      26793         root   15u     unix 0xffff880a9d9bc840         0t0 3248768164 /var/lib/sss/pipes/private/sbus-monitor
sssd      26793         root   16u     unix 0xffff880040a32f00         0t0 3248768165 /var/lib/sss/pipes/private/sbus-monitor
sssd_be   26794         root   15u     unix 0xffff8808cc064200         0t0 3248767368 /var/lib/sss/pipes/private/sbus-dp_default.26794
sssd_be   26794         root   16u     unix 0xffff880a9d9bd880         0t0 3248763661 /var/lib/sss/pipes/private/sbus-dp_default.26794
sssd_be   26794         root   17u     unix 0xffff8809841b4480         0t0 3248763662 /var/lib/sss/pipes/private/sbus-dp_default.26794
sssd_nss  26795         root   16u     unix 0xffff880a9d9bd200         0t0 3248751954 /var/lib/sss/pipes/nss
sssd_pam  26796         root   16u     unix 0xffff880859e26180         0t0 3248774325 /var/lib/sss/pipes/pam
sssd_pam  26796         root   17u     unix 0xffff880859e27b80         0t0 3248774326 /var/lib/sss/pipes/private/pam

Also, /var/log/secure containes multiple entries of 

sshd[9177]: pam_succeed_if(sshd:auth): error retrieving information about user
su: pam_sss(su-l:session): Request to sssd failed. Connection refuse
crond[29568]: pam_sss(crond:session): Request to sssd failed. Connection refused

Additionally, one of the first things I noticed was that the /var/log/messages file contained no data. Both it and /var/log/sssd/ logs seem to have stopped collecting around 9:03 this morning, /var/log/secure kept accumulating data without issue. Restarting syslog fixed the issue for mesages, but sssd logs are still not functioning.

Last thing I noticed dmesg is filled up with messages like audit: backlog limit exceeded audit: audit_backlog=322 > audit_backlog_limit=320 and audit_log_start: 122 callbacks suppressed. I assumed these are from when syslog wasn't working working properly, but haven't verified that, yet.

I'm still researching into this and hope I'll find something, but more than welcome any suggestions and feedback people are willing to provide.

Thanks a lot!

-Omni

omnivir
  • 99
  • 1
  • 1
  • 3
  • Do you have multiple domains configured in your sssd.conf? – HTTP500 Feb 23 '15 at 23:52
  • Just one domain is specified in the conf file. the contents of which are identical between all Linux bservers tired into the ldap domain and this issue has only been seen on the one server. – omnivir Feb 24 '15 at 05:51
  • Any kernel oopses or similar? Any disk errors? – Dan Feb 24 '15 at 08:14
  • I'm still running through some of the older message files that were archived, but I haven't run across anything that screams out as a kernel panic or whatnot. Each of the message files I've looked at so far have multiple entries similar to the one I mention above for dmesg, but I'm not seeing any panics or anything. dmesg is filled with nothing but those audit_backlog messages, so that hasn't been too helpful. – omnivir Feb 24 '15 at 12:42
  • Well I was wondering if you had dropped an LDAP domain maybe that confused SSSD. Maybe you could try purging the cache? i.e. "sss_cache -E" – HTTP500 Feb 24 '15 at 17:08
  • Not sure if this is still relevant but the reason kill -9 won work is because the processes are in "uninterruptible sleep" (see ps listing). So to stop the processes you need to find out what is blocking them. Things to check: is SELinux on? If yes: any AVC denials? (running restorecon migt help). Are there any remote file systems involved? Other file system issues that might prevent SSSD reading/writing? Network issues between this host and the directory server? (routing, firewall) Finally - assuming this is a Red Hat system - I recommend opening a support case if you haven't already. – Bram Oct 14 '15 at 06:46

1 Answers1

0

I have run into issues with pam_ldap and sssd running on the same system. If you want to stop using sssd with pam you should make sure that you run:

pam-config -a --ldap

which will add LDAP to pam and then run:

pam-config -d --sss

which will delete the sssd setting in pam related config files in /etc/pam.d/

To make sure that sss is not used you may also want to check that nsswitch.conf has ldap in the correct places (or at the very least does not have sss). Here is an /etc/nsswitch.conf with sss enabled:

#
# /etc/nsswitch.conf
#
# An example Name Service Switch config file. This file should be
# sorted with the most-used services at the beginning.
#
# The entry '[NOTFOUND=return]' means that the search for an
# entry should stop if the search in the previous entry turned
# up nothing. Note that if the search failed due to some other reason
# (like no NIS server responding) then the search continues with the
# next entry. 
#  
# Legal entries are:
#
#       compat                  Use compatibility setup
#       nisplus                 Use NIS+ (NIS version 3)
#       nis                     Use NIS (NIS version 2), also called YP
#       dns                     Use DNS (Domain Name Service)
#       files                   Use the local files
#       [NOTFOUND=return]       Stop searching if not found so far  
#
# For more information, please read the nsswitch.conf.5 manual page.
#

passwd: compat sss
group:  compat sss

hosts:  files mdns_minimal [NOTFOUND=return] dns
networks:   files dns

services:   files
protocols:  files
rpc:    files
ethers: files
netmasks:   files
netgroup:   files
publickey:  files

bootparams: files
automount:  files
aliases:    files
passwd_compat:  files
group_compat:   files

This file has sss disabled and ldap enabled:

passwd: compat ldap
group:  compat ldap

hosts:  files mdns_minimal [NOTFOUND=return] dns
networks:   files dns

services:   files
protocols:  files
rpc:    files
ethers: files
netmasks:   files
netgroup:   files
publickey:  files

bootparams: files
automount:  files
aliases:    files
passwd_compat:  files
group_compat:   files
Matt
  • 2,751
  • 1
  • 14
  • 20