Why requires SSH public key authentication the passphrase again after 2k sessions?

Question

I recently started the replacement of my backup storage. The Mac Mini with a WD MyBook is being replaced by a Synology DiskStation. On of the backup scripts i have been using to synchronize my external HDD with VMs stopped working properly after relocation the backup destination to the Synology. The script basically starts a rsync process in a SSH session per file. Details of the script can be found here; http://www.reddipped.com/2016/07/speeding-vm-backups-using-rsync/

To be able to run the script without having to type in a password for each SSH session, public key authentication has been configured.

Client configuration (Mac OS X Sierra)

• Create the private/public keypair

    ssh-keygen -t rsa -f ~/.ssh/id_rsa -C "rsync"

• Fill in random passphrase, e.g. "rsync protocol"

• Copy the public key to the users home directory on the synology

  ssh Peter\ van\ Nes@newyork 'mkdir ~/.ssh'
  cat ~/.ssh/id_rsa.pub | ssh Peter\ van\ Nes@newyork 'cat - >> ~/.ssh/authorized_keys'
  ssh Peter\ van\ Nes@newyork 'chmod 644 .ssh/authorized_keys'
  ssh Peter\ van\ Nes@newyork 'chmod 755 ~ ~/.ssh' 
  

Server (Synology)

• Open SSH session to Synology

  ssh "Peter van Nes"@newyork
  

• Edit sshd_config

  vi /etc/ssh/sshd_config   
  # Enable RSAAuthentication yes     
  # Enable PubkeyAuthentication yes
  # Enable AuthorizedKeysFile .ssh/authorized_keys
  # Enable ChallengeResponseAuthentication yes
  # Add MaxSessions 50
  # Add MaxStartups 50:10:100
  

• Restart the services

  # restart ssh-shell
  synoservicectl --restart ssh-shell
  # restart rsync daemon
  synoservicectl --restart rsyncd   
  

After completing these steps i am able to logon using a public key successfully. When running the script it initially runs as expected, only after a few thousand files the passphrase for the key has to be entered again.

I did break the issue down to this little script which reproduces the issue for me.

#!/bin/bash

sessions=0
while true
do
    sessions=$(($sessions + 1))
    echo "Session " $sessions
    ssh peter@newyork -x "ls -al"
done 

After a few thousand sequential SSH sessions the passphrase for the public key is requested again. The sequential successful number of sessions differs per run. This can be 2102 but also 5000+.

Session  2101
total 4
drwxr-xr-x  1 peter users   24 Oct 15 12:46 .
drwxrwxrwx+ 1 root  root    92 Oct 16 22:29 ..
drwxr-xr-x  1 peter users   30 Oct 15 12:46 .ssh
-rwxrwxrwx+ 1 peter users 1239 Oct 15 12:46 .viminfo
Session  2102
total 4
drwxr-xr-x  1 peter users   24 Oct 15 12:46 .
drwxrwxrwx+ 1 root  root    92 Oct 16 22:29 ..
drwxr-xr-x  1 peter users   30 Oct 15 12:46 .ssh
-rwxrwxrwx+ 1 peter users 1239 Oct 15 12:46 .viminfo
Session  2103
Enter passphrase for key '/Users/petervannes/.ssh/id_rsa': 

When running ssh with the -vvv argument does not give any hints, also the auth.log on the Synology does report any authentications errors.

Any idea how to debug or fix this?

Answer 1

Fixed it by adding the certificates to the keychain using ssh-add -K.

#!/bin/bash

sessions=0

ssh-add -K

while true
do
    sessions=$(($sessions + 1))
    echo "Session " $sessions
    ssh peter@newyork -x "ls -al"
done