31

It's a sort of regular occurrence that, after SSHing to a server (or even opening a terminal on my Mac), the login banner prints immediately but it takes ~10 seconds to a minute for the shell prompt to appear. After that, performance is fine and network latency is not unusual.

This doesn't seem like a computationally difficult, memory-intensive, or IO-heavy task. What's it doing with all those billions of CPU cycles?

jacobbaer
  • 473
  • 1
  • 4
  • 5
  • 9
    `ssh -v -v -v` and `ur_shell -x` might be prudent debugging steps. – thrig Sep 15 '15 at 17:26
  • 2
    Is there a lot of lines in your `.bash_history`? – kasperd Sep 15 '15 at 17:37
  • 2
    Look at your .profile and related files (sorry, not sure which shell you're using) and remove temporarily to see if that improves things. You probably have a command(s) that are timing out. – TheFiddlerWins Sep 15 '15 at 17:45
  • Does it take that long if you open a terminal window locally? – Moby Disk Sep 15 '15 at 20:42
  • 3
    Alternatively to Moby Disk's suggestion, is it still equally slow if you start the shell itself from within the shell? If it's slow when (re)connecting but fast when starting a second shell from within an existing, connected session, that will tell you that it's not the shell itself that is causing the slowness; if it is equally slow in the two situations, then something the shell does on startup is taking a lot of time. Either way you learn some about which aspect of "from zero to shell prompt" is slow. – user Sep 15 '15 at 20:55
  • 2
    Often this is GSSAPI authentication trying to occur (which, if you're not a Kerberos shop, is likely completely useless). Other times, it's DNS reverse lookups. – Charles Duffy Sep 15 '15 at 22:32
  • Do you have bash_completion installed ? If so, change your profile and related initialization files to avoid loading it (or update it to the latest version). – xryl669 Sep 16 '15 at 14:09
  • 1
    "Opening a terminal on your mac"? Are you talking about running xterm on the server, and having it display on your Mac desktop with SSH X11 forwarding? Or are you saying that new terminal windows are slow on your local desktop, as a separate issue from sshing to your server? – Peter Cordes Sep 16 '15 at 18:00
  • Another likely culprit is the tab completion logic -- here, a lot of scripts are executed for each shell to set up the completion tables. – Simon Richter Oct 08 '21 at 10:44

8 Answers8

33

Quite a few things could be happening here. You can find most of the answers in your shell's manual, but those are usually incredibly long and oblique, so...

Chances are your problem boils down to one of a few things.

If your profile or bashrc have expensive things, consider trimming them back.

If your profile or bashrc use a reverse DNS lookup (to set the prompt or something), fix DNS or use hostname instead.

Shells open a lot of files, among other things, while initialising. If system load is high, it will often show up here.

If the banner is pre-authentication, it might also actually be authentication (pam, LDAP, etc.) that is slow.

It might be none of these things, though. A surprising amount of stuff happens right before displaying the prompt!

Falcon Momot
  • 25,244
  • 15
  • 63
  • 92
  • 1
    Another one I've seen block for a substantial while is attempted GSSAPI authentication (if not turned off on either the SSH client or server config). Using a full-system tracing tool such as sysdig is probably the best silver bullet available to get to the bottom of this kind of cross-concern issue. – Charles Duffy Sep 15 '15 at 22:33
  • +1 Thorough answer. [This site](http://blog.flowblok.id.au/2013-02/shell-startup-scripts.html) has a nice flowchart for logins, which files they execute/source, and other info. – Tim S. Sep 15 '15 at 23:22
  • 15
    +1. 99% of the time for me it's the reverse DNS lookup. – mpontillo Sep 15 '15 at 23:38
  • @Mike: same here - I would recommend to start with this one as it is easy to fix. It was 100% in my case. – WoJ Sep 16 '15 at 15:19
22

It's probably either waiting for DNS or trying to authenticate via LDAP or such.

Try to add UseDNS no to /etc/ssh/sshd_config

If it also does it on local logons, check if any LDAP servers or DNS servers you have configured are slow or unresponsive.

sCiphre
  • 463
  • 3
  • 6
6

One possibility (covered by other answers) is that the process of setting up the SSH session itself is where the time is lost.

Another alternative is that your shell startup scripts running on the remote machine after SSH session establishment have something that is taking a long time (perhaps attempting to access some broken network mount). You can debug this second possibility as follows:

Temporarily add the following to the top of your ~/.bash_profile:

set -x
PS4='+ $(date "+%s.%N")\011 '

The set -x turns on some debugging for every shell command executed. The PS4 variable controls how that debugging is presented - specifically in this case we use date to add timestamps.

You may then analyze the timestamps of the debugging output to see which commands in your startup scripts are taking too long.

Digital Trauma
  • 203
  • 1
  • 10
4

If it's an Ubuntu server, the default login setup checks if any packages are updatable every time a login shell runs. If the package lists aren't in disk cache, this can take a second or two even on a fast idle desktop.

$ ssh localhost 
Welcome to Ubuntu 15.04 (GNU/Linux 3.19.0-26-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

*** System restart required ***
Last login: Sat Sep 12 01:38:38 2015 from localhost

To generate that "restart required" message, it had to check that the currently running kernel is not the default kernel currently installed. (i.e. there was a kernel and I didn't reboot yet.) It will also print a count of security updates available, if there are any.

I think this is the major slowdown in logging in to Ubuntu that's been introduced recently.

If not that, then your ~/.bash_profile / ~/.bashrc might be the problem.

Have you tried logging in to the server from itself (ssh localhost)? Or logging in a 2nd time right away? (To see if it's a lot faster when stuff is cached.)

Peter Cordes
  • 457
  • 4
  • 10
2

This is in most cases a timeout of a DNS request.

Cause: The server attempts a reverse DNS lookup using the IP address of the client, and does not get a reply. If A connects to B, B tries to convert the IP address of A to a name.

Workaround: Enter the IP-Address and name of the client to the hosts file of the server.

Solution: make all hosts known to the DNS server.

Klaus Hartnegg
  • 331
  • 1
  • 7
0

Learned alot from this thread. In my case the offender was nvm:

export NVM_DIR="$HOME/.nvm"
[ -s "/usr/local/opt/nvm/nvm.sh" ] && . "/usr/local/opt/nvm/nvm.sh"  # This loads nvm
[ -s "/usr/local/opt/nvm/etc/bash_completion" ] && . "/usr/local/opt/nvm/etc/bash_completion"  # This loads nvm bash_completion
Paul
  • 101
0

for future visitors, i had miniconda on my machine and its .bashrc entry was the one causing my delayed shell prompt. i no longed needed the conda so removed its entry from .bashrc that looked kike this

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/<YOUR_USER>/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/<YOUR_USER>/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/home/<YOUR_USER>/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/<YOUR_USER>/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<
Rami Amro Ahmed
  • 101
0

a common issue i have is, that the server is Reachable by IPv4 and IPv6 and due the fact that IPv6 will be prefered even if its unsupported to the internet of the LAN, it tries to connect.

On Putty you can solve this, that you use instead AUTO, prefer ipv4 as default

this behaviour seems a windows issue mostly, but found also on some linux boxes also

djdomi
  • 1,599
  • 3
  • 12
  • 19