The actual problem was solved by changing the port number but I don't believe I understood why it was happening in the first place and so I'd like to get to the bottom of it for next time, especially because hostile action cannot be ruled out.
I admin a corporate Ubuntu 14.04 VPS in AWS EC2. The server ssh port is open to 0.0.0.0/0 because this server is the gateway to a VPC from whatever insecure location I'm in (business hotel, airport, starbucks etc). SSH is obviously configured for non-root, keys-only connections. Because the ssh port is world visible, I changed the port to 50022 reasoning that broad sweeps of the internet would mostly miss it and it would only be picked up in targeted a port probe, reducing CPU cycles and log entries denying fruitless dictionary attacks as well as network traffic.
For whatever reason, I would routinely but intermittently fail to connect on ssh to this server. Failure was much more common than success but I would succeed occasionally without doing anything different.
I have successfully used this port number on other non-EC2 servers so it's probably not a general problem with the port.
Changing the port number just now to another high-numbered human-random port seems to have resolved it.
I'm in China when I have this problem so I can't rule out firewall shenanigans or even some NSA-style automated mitm probing for key/certificate collection or VPN blocking. The server does host VPN services so interference from those could potentially be related though this is unlikely because I also use non-EC2 VPSs with VPNs that do not have problems with ssh on this port.
Is there anything special about port 50022?
