9

I'm having an issue on CentOS 6.7 where SSH logins are 1s more than any non-6.7 machines on this network (e.g. 7.2, 5.11). Running debugging on the client side showed the hang at "Entering interactive session".

The command I'm using to base this test on is time ssh <host> true from my laptop, using SSH Keys.

The two things I've already checked/modified are UseDNS and GSSAPIAuthentication, and both are disabled.

I started a separate daemon on a different port with debugging on, and have found where the short hang occurs:

debug1: SELinux support enabled
debug3: ssh_selinux_setup_exec_context: setting execution context

{1s hang}

debug3: ssh_selinux_setup_exec_context: done

SELinux is set to 'permissive'. I'm not sure why it would even bother with "setting contexts". Is there any way to have these two get along better without disabling SELinux altogether? I realize 1s isn't much, but I use this particular machine as an SSH gateway to hosts with IP whitelists (this is a static IP machine), and it adds up throughout the day.

After running an strace, the hang is a little more granular:

22:16:05.445032 open("/selinux/user", O_RDWR|O_LARGEFILE) = 4 <0.000090>
22:16:05.445235 write(4, "unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 unconfined_u", 56) = 56 <0.334742>
22:16:05.780128 read(4, "18\0unconfined_u:system_r:prelink_mask_t:s0-s0:c0.c1023\0unconfined_u:system_r:unconfined_mount_t:s0-s0:c0.c1023\0unconfined_u:system_r:abrt_helper_t:s0-s0:c0.c1023\0unconfined_u:system_r:oddjob_mkhomedir_t:s0-s0:c0.c1023\0unconfined_u:system_r:unconfined_notrans_t:s0-s0:c0.c1023\0unconfined_u:system_r:unconfined_execmem_t:s0-s0:c0.c1023\0unconfined_u:system_r:unconfined_java_t:s0-s0:c0.c1023\0unconfined_u:system_r:unconfined_mono_t:s0-s0:c0.c1023\0unconfined_u:system_r:chkpwd_t:s0-s0:c0.c1023\0unconfined_u:system_r:passwd_t:s0-s0:c0.c1023\0unconfined_u:system_r:updpwd_t:s0-s0:c0.c1023\0unconfined_u:system_r:mount_t:s0-s0:c0.c1023\0unconfined_u:system_r:rssh_t:s0-s0:c0.c1023\0unconfined_u:system_r:xauth_t:s0-s0:c0.c1023\0unconfined_u:system_r:unconfined_t:s0-s0:c0.c1023\0unconfined_u:system_r:openshift_t:s0-s0:c0.c1023\0unconfined_u:unconfined_r:oddjob_mkhomedir_t:s0-s0:c0.c1023\0unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023\0", 4095) = 929 <0.000079>

Writing to /selinux/user takes 350ms alone.

Update 1 - Things I've since tried:

  • Updating. Many of the boxes required updates. This hasn't had any material effect on login times.
  • semodule -d unconfined - This has had a positive effect, reducing the login times by roughly 500ms. However, my C7 and C5.11 boxes (which I've since (re)learned have SELinux disabled) average ~525ms
  • Comparing my C6.7 machines - Turns out the 64bit boxes are faster than my 32-bit boxes. Maybe this is a 'duh' moment for some, but since none of these boxes are particularly taxed, I wasn't expecting a 100-300ms variance. I was able to break the 1s mark (850ms) on one of the 64bit hosts. The lowest on 32bit has been 1.085s

Am I asking too much? Is 10-1200ms an acceptable login time for selinux-permissive machines? Curious what other people's benchmarks are.

Morgon
  • 203
  • 1
  • 6
  • Check your system logs. – Michael Hampton Jan 17 '16 at 15:20
  • I have... Unfortunately, nothing noteworthy in `messages` or `secure` – Morgon Jan 17 '16 at 15:22
  • @Morgon: Have you ckecked if you're running something like "fail2ban" - http://www.fail2ban.org/ - in that machine and/or if you have set up "iptables" in some way that may be doing some kind of "rate limiting" to incoming SSH connections depending on source IP address(es) ? – ricmarques Jan 17 '16 at 16:25
  • @ricmarques - F2B is not installed. I'm not sure how to rate limit with iptables alone, so I'm going to have to say no to your second question, as well. :) – Morgon Jan 17 '16 at 16:28
  • 4
    If logs aren't helping, you should probably use `strace` and see what call it is hanging on. – Andrew B Jan 17 '16 at 18:27
  • 1
    Thanks, @AndrewB. I've posted it here: http://pastebin.com/raw/3c08tcMd. if you search for "/selinux/user", you'll see the 1s hangtime while trying to write to that file. -- Going down that path, I found a marginally-related post about disabling 'unconfined' domains (https://bugzilla.redhat.com/show_bug.cgi?id=811656#c9). This does shave ~.5 seconds from my logins, but still .5 seconds slower than Cent7/FC4. Still looking to eke out that last bit of performance, if possible. – Morgon Jan 17 '16 at 20:30
  • Looks like you're losing nearly a full second on a write to `/selinux/user`, but unfortunately I'm no selinux expert. Someone else will have to speak up. – Andrew B Jan 17 '16 at 21:29
  • Do you see a lot of misses in `avcstat` relative to hits? – Aaron Jan 17 '16 at 23:12
  • @Aaron - Yeah, about 23%. Running `avcstat` with a 1 second interval, there were 850+ 'misses' among ~3800 lookups between the two seconds that I tested a login connection. Not being very selinux-savvy, what does this indicate? – Morgon Jan 18 '16 at 00:57
  • There are many possibilities, but it could be that something else on the box is taxing the avc cache and causing a ripple effect (your ssh slowness). Entirely speculation on my part. You might take a look at the value of `/sys/fs/selinux/avc/cache_threshold` and then double it. e.g. `echo 1024 > /sys/fs/selinux/avc/cache_threshold` and see if the slowness is reduced. Anything beyond this point is speculation without having more understanding of the workloads on that system. – Aaron Jan 18 '16 at 01:08
  • Have any customizations occured in pam? Have you looked at `authconfig --test` and does anything trigger memories of a tweak someone may have made? Do you know when the behavior started and can you correlate that to process running time, or file modification times in /etc? Also, I see pam is calling oddjob. Are your home directories nfs mounted by chance? – Aaron Jan 18 '16 at 01:15
  • One last question: Do you have UsePAM set to yes in your sshd_config? SELinux depends on PAM being enabled in ssh or it will hang for a bit. – Aaron Jan 18 '16 at 01:23
  • @Aaron - Hrm. What if I don't have a `/sys/fs/selinux/avc/cache_threshold` file? There's an `/selinux/avc/cache_threshold`, but I cannot write to it (by design, apparently). This particular machine handles mail and DNS; it really shouldn't be taxed that badly. Plus, I have several other machines that are doing absolutely nothing and exhibit the same behavior (including `avcstat` numbers) – Morgon Jan 18 '16 at 01:26
  • I did have `UsePAM` off as a test. Turning it on seems to increase login times by 150-200ms on average. No customizations in PAM. This has been going on as long as Cent 6.7 has been installed, as far as I know. ALL of my 6.7 machines exhibit this behavior, and I definitely wouldn't have made customizations to all of them, since I haven't been using every machine. There are no NFS mounts. – Morgon Jan 18 '16 at 01:27
  • [Continuing discussion in chat](http://chat.stackexchange.com/rooms/34455/discussion-between-morgon-and-aaron). – Morgon Jan 18 '16 at 01:32
  • How much entropy do you have available? – ptman Aug 25 '17 at 08:34
  • What is the output of `avcstat`? – Aaron Jul 20 '18 at 20:34
  • Did you ever figure this out? – oxr463 Jun 05 '19 at 16:00
  • Sadly not; I simply ended up upgrading those machines, which of course fixed it in its own way. – Morgon Jun 10 '19 at 23:24

1 Answers1

1

This may not be related, but selinux related slowness has been report for IPA-based accounts here: https://access.redhat.com/discussions/3499951

The solution for that case is:

I looked into SSSD and IPA user forums and since my systems do not use SELINUX, I went ahead and added this to the domain section in the sssd configuration:

selinux_provider=none

After this the login process is immediate, as expected.

chutz
  • 7,888
  • 1
  • 29
  • 59