SSH/SFTP connections fails silently on OSX 10.8.2 - ssh-agent issue

Question

I'm trying to login to a remote machine with SSH or SFTP.

when I try ssh u-indgo@ssh1.eu1.frbit.com the CLI just won't respond. I get an empty new line, in which I can type characters, but nothing more.
when I try to connect with SFTP using the same credentials (I use Transmit as my SFTP client) it just hangs forever and doesn't connect.

No errors. No response.

The problem isn't specific to frbit.com and persists with any other server I try to connect to.

ssh client debugging

Running the ssh client with the -vv flag I got the following output:

debug1: Reading configuration data /Users/matanya/.ssh/config
debug1: Reading configuration data /usr/local/Cellar/openssh/6.1p1/etc/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to ssh1.eu1.frbit.com [46.137.57.195] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: identity file /Users/matanya/.ssh/id_rsa type 1
debug1: identity file /Users/matanya/.ssh/id_rsa-cert type -1
debug1: identity file /Users/matanya/.ssh/id_dsa type 2
debug1: identity file /Users/matanya/.ssh/id_dsa-cert type -1
debug1: identity file /Users/matanya/.ssh/id_ecdsa type -1
debug1: identity file /Users/matanya/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.5p1
debug1: match: OpenSSH_5.5p1 pat OpenSSH_5*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.1
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 140/256
debug2: bits set: 543/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 31:4c:71:e0:56:14:04:0d:c7:b2:6c:fc:8a:42:33:2e
debug1: Host 'ssh1.eu1.frbit.com' is known and matches the RSA host key.
debug1: Found key in /Users/matanya/.ssh/known_hosts:2
debug2: bits set: 513/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received

ssh-agent debugging

UPDATE: going through my local (ssh client machine) system.log I found the following:

Mar  6 10:28:17 matanyas-imac com.apple.launchd.peruser.501[235] (org.openbsd.ssh-agent[574]): Exited with code: 1
Mar  6 10:28:17 matanyas-imac com.apple.launchd.peruser.501[235] (org.openbsd.ssh-agent): Throttling respawn: Will start in 10 seconds
Mar  6 10:28:27 matanyas-imac com.apple.launchd.peruser.501[235] (org.openbsd.ssh-agent[575]): Exited with code: 1
Mar  6 10:28:27 matanyas-imac com.apple.launchd.peruser.501[235] (org.openbsd.ssh-agent): Throttling respawn: Will start in 10 seconds

What does Code 1 stand for?

UPDATE: I found the file that launchd has problems with at System/Library/LaunchAgents/org.openbsd.ssh-agent.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.openbsd.ssh-agent</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/ssh-agent</string>
        <string>-l</string>
    </array>
    <key>ServiceIPC</key>
    <true/>
    <key>Sockets</key>
    <dict>
        <key>Listeners</key>
        <dict>
            <key>SecureSocketWithKey</key>
            <string>SSH_AUTH_SOCK</string>
        </dict>
    </dict>
        <key>EnableTransactions</key>
        <true/>
</dict>
</plist>

When I run /usr/bin/ssh-agent I get:

SSH_AUTH_SOCK=/var/folders/pg/1g6_hnwx47bgqv5vcm1lq18h0000gn/T//ssh-01WuaHF32SlV/agent.2145; export SSH_AUTH_SOCK;
SSH_AGENT_PID=2146; export SSH_AGENT_PID;
echo Agent pid 2146;

as for the -l flag (<string>-l</string>) there is no such flag on my version of ssh-agent. Outputs:

ssh-agent: illegal option -- l

ps aux | grep ssh outputs:

matanya         1121   0.0  0.0  2441136   3280   ??  S     1:53PM   0:00.01 ssh -oNumberOfPasswordPrompts 1 -2 -lu-indgo -s ssh1.eu1.frbit.com sftp
matanya         1116   0.0  0.0  2441136   3280   ??  S     1:52PM   0:00.01 ssh -oNumberOfPasswordPrompts 1 -2 -lu-indgo -s ssh1.eu1.frbit.com sftp
matanya         1101   0.0  0.0  2441136   3280   ??  S     1:51PM   0:00.01 ssh -oNumberOfPasswordPrompts 1 -2 -lu-indgo -s ssh1.eu1.frbit.com sftp
matanya         1095   0.0  0.0  2441136   3280   ??  S     1:50PM   0:00.01 ssh -oNumberOfPasswordPrompts 1 -2 -lu-indgo -s ssh1.eu1.frbit.com sftp
matanya         1084   0.0  0.0  2441136   3280   ??  S     1:50PM   0:00.01 ssh -oNumberOfPasswordPrompts 1 -2 -lu-indgo -s ssh1.eu1.frbit.com sftp
matanya         1593   0.0  0.0  2439184   2092 s000  S+    2:36PM   0:00.00 grep ssh

SSH version: OpenSSH_5.8p2, OpenSSL 0.9.8r 8 Feb 2011

UPDATE: I've discovered that it doesn't matter with which user I initially login on system boot - be it my own or the root - ssh won't work until i explicitly switch user in the terminal (su -or su matanya)

UPDATE:

I checked the code signatures. Ran: codesign -vv /usr/bin/ssh-agent:

received:

/usr/bin/ssh-agent: code object is not signed at all
In architecture: x86_64

Should be:

/usr/bin/ssh-agent: valid on disk
/usr/bin/ssh-agent: satisfies its Designated Requirement

UPDATE:

When I run :

eval `ssh-agent`
ssh-add

I can login with ssh.

possible duplicate of [ssh hangs without password prompt -- works in root or other accounts](http://serverfault.com/questions/57509/ssh-hangs-without-password-prompt-works-in-root-or-other-accounts) — quanta, Mar 06 '13 at 07:28
I forgot to mention that it fails generally, with all different IP's and usernames — Matanya, Mar 06 '13 at 07:51
If it falls generally for everyone, then you really **really** need to start debugging at the server side, not the client side. — Jenny D, Mar 06 '13 at 07:53
Troubleshooting almost always starts with looking in the logs. In this case those are probably /var/log/auth.log, maybe /var/log/system.log or /var/log/messages, depending on your syslog configuration. — Jenny D, Mar 06 '13 at 08:17
When you say "The problem isn't specific to this host", It's not clear what's being tested. I would say three ssh clients (a, b, and c) would be needed to connect to two sshd (x and y) and you could have **a** be your Mac client, **b** be another Mac 10.8.2 (or at least a new user on that Mac) and **c** be a different client. **x** would be the problem remote `ssh1.eu1.frbit.com` and **y** might be a different remote server that works for you from **a** - the `matanyas-imac` Only then would you be able to isolate which combination might be causing the failure. — bmike, Mar 07 '13 at 13:46

Lætitia · Answer 1 · 2013-03-12T09:41:44.417

12

Reason of the silent failing when connecting

Your system.log errors show you have an issue with your ssh-agent running locally on your iMac. For some reason it doesn't run even if launchd tries to restart it.

When you try to connect using any ssh client (CLI or Transmit) they try to use ssh-agent but they cannot connect to it as it's not running. Hence their waiting without output nor input.

I'm not sure what prevents your ssh-agent from running. However, to run your ssh client on the CLI and make it connect to your servers, you can try the following:

unset SSH_AUTH_SOCK
ssh u-indgo@ssh1.eu1.frbit.com
# (you'll then be asked for you pass phrase if you use one)

You can even try to launch Transmit from the same Terminal window:

open /Applications/Transmit.app

About ssh-agent debugging

If ssh-agent -l tells you the -l option is illegal, it means it's not the original Apple ssh-agent that your system is trying to run (-l is an Apple undocumented feature). The replacing ssh-agent is making launchd unhappy. This blog post might have some explanations why.

If you have third party ssh tools (coming from brew, macports or other channels), I'd recommend you move them out of the way or you upgrade them (provided they are launchd capable, i.e.: the -l option exists). A working ssh-agent invocation should answer something like:

antoine@amarante:~$ /usr/bin/ssh-agent -l
launch_msg: Operation not permitted

It is also a good idea to check you don't start ssh-agent from other places like .bashrc or other session startup scripts. Having multiple, and possibly different, ssh-agent running at the same time is potentially a source of problem.

edited Mar 12 '13 at 09:41

answered Mar 08 '13 at 11:09

Lætitia

2,085
22
33

Thx. Unsetting SSH_AUTH_SOCK did work for me, as well as switching to root user. – Matanya Mar 08 '13 at 11:28
The ssh-agent is running for your own user only, that's why switching to the root user also works. You definitely need to look at your ssh-agent configuration as it is the one that prevents you from connecting to your server. – Lætitia Mar 08 '13 at 11:35
Which version of OSX are you using? The `org.openbsd.ssh-agent.plist` looks the same as mine, there is nothing wrong there. Do you see anything else in your logs (in Console.app)? Don't you have multiple `ssh-agent` processes running at the same time? – Lætitia Mar 08 '13 at 11:56
I'm using the latest 10.8.2 OSX Lion. How do I test for multiple `ssh-agent` processes? – Matanya Mar 08 '13 at 12:31
What does `ps aux | grep ssh` outputs? On my 10.8.2 when I run `ssh-agent -l` I get a `launch_msg: Operation not permitted` error. If you're not getting that (but instead have the illegal option error), it's because the ssh-agent your system is trying to run is not `launchd` compatible. Do you have third party ssh tools installed? brew, macports, fink installed? – Lætitia Mar 08 '13 at 12:35
i have brew and macports installed. I've uninstalled both brew and macports openssh. I've pasted the output for `ps aux | grep ssh` in my question – Matanya Mar 08 '13 at 12:39
It would be best to also restart your computer after having them un-installed (or at least log out of your session). That way ssh-agent will have a better chance to launch. Look if you still have the launchd errors in your system.log after that reboot. – Lætitia Mar 08 '13 at 12:42
i've uninstalled them a few days ago, and have rebooted my machine since – Matanya Mar 08 '13 at 12:45
let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/7817/discussion-between-tonin-and-matanya) – Lætitia Mar 08 '13 at 13:14

score 1 · Answer 2 · edited Aug 10 '21 at 23:05

Would you please to check SSH connection with other program like Cyberduck?

Also I've found solution where you need to check launch agents at following locations:

/Macintosh HD/Library/LaunchAgents/
/Macintosh HD/Library/LaunchDaemons/
/username/Library/LaunchAgents/
/username/Library/LaunchDaemons/

and then check missing executables or files without executable flag toggled.

Mine OpenSSH local version is OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011

So you can also try OpenSSH from macports or brew. I personally prefer macports then brew if I need anything not in OS X by default.

UPDATE:

Try to run ssh -a u-indgo@ssh1.eu1.frbit.com, same as above but with disabled agent forwarding
check if your Keychain Access keys are correct
check if your directory ~/.ssh has correct permissions (0600)
check if your keys are correct.
try to run "source `ssh-agent`" before executing ssh command

UPDATE2:

On my system (OS X 10.8) org.openbsd.ssh-agent.plist looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>org.openbsd.ssh-agent</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/ssh-agent</string>
        <string>-l</string>
    </array>
    <key>ServiceIPC</key>
    <true/>
    <key>Sockets</key>
    <dict>
        <key>Listeners</key>
    <dict>
            <key>SecureSocketWithKey</key>
            <string>SSH_AUTH_SOCK</string>
        </dict>
    </dict>
        <key>EnableTransactions</key>
        <true/>
</dict>
</plist>

Also I ses this:

$ /usr/bin/ssh-agent -l
launch_msg: Operation not permitted
$ shasum -a 256 /usr/bin/ssh-agent 
e21e2f23819b60f6288edda97427d98413c1bb737d49d313e2857f058627aab6  /usr/bin/ssh-agent

Tell me please information related to permissions, your ~/.ssh/confing and Keychain Access. — Eir Nym, Mar 07 '13 at 16:33
/usr/local/Cellar/php54/5.4.11/bin:/usr/local/sbin:/opt/local/bin:/opt/local/sbin:/usr/local/share:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin — Matanya, Mar 08 '13 at 09:47
Can you double check you path & library path to be sure nothing had been hijacked, as it seems from your answer? May be in first remove anything from your path except /bin:/sbin:/usr/bin:/usr/sbin — Eir Nym, Mar 08 '13 at 10:16

SSH/SFTP connections fails silently on OSX 10.8.2 - ssh-agent issue

ssh client debugging

ssh-agent debugging

2 Answers2

Reason of the silent failing when connecting

About ssh-agent debugging