5

I have set up a few 100 embedded boxes to contact HQ by opening up reverse ssh tunnels, each under a new port. This is mostly working fine, but today I encountered a problem with using the tunnel through a low bandwidth (or low quality?) GPRS connection.

The remote machine opening up the tunnel is connected to the inter via a (so far unknown) 3G router which probably only has a GPRS, an EDGE connection at best.

Logged in to my machine I can see the incoming ssh connection on it's port 1234:

me@machine:~$ sudo nmap -sS -p 1234 --open localhost

Starting Nmap 5.21 ( http://nmap.org ) at 2014-11-27 15:27 CET
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000070s latency).
Hostname localhost resolves to 2 IPs. Only scanned 127.0.0.1
PORT     STATE SERVICE
1234/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds
me@machine:~$

Now, trying to open the ssh connection I get a Connection timed out error:

me@machine:~$ ssh -vp 1234 localhost
OpenSSH_5.9p1 Debian-5ubuntu1.4, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to localhost [::1] port 1234.
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: identity file /home/cts/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-4096
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-4096
debug1: identity file /home/cts/.ssh/id_rsa-cert type -1
debug1: identity file /home/cts/.ssh/id_dsa type -1
debug1: identity file /home/cts/.ssh/id_dsa-cert type -1
debug1: identity file /home/cts/.ssh/id_ecdsa type -1
debug1: identity file /home/cts/.ssh/id_ecdsa-cert type -1
Connection timed out during banner exchange
me@machine:~$

Other ports work fine, as did this one when it was (verified) connected to a 3G network.

I tried something that sometimes helps when doing this over satellite connections - ssh -o "ConnectTimeout 99" -o "ServerAliveCountMax 5" -vp 1234 localhost but that didn't help either.

I assume this has something to do with either
a) the wireless provider filtering something on his GPRS network he does not on his 3G network
or
b) the bad latency of the GPRS connection f***ing up my tunnel.

Anyone have an idea of how to tackle this situation or get a better understanding of what is going on (or, rather, not going on) here? Adding more vs to the command doesn't show any more debug output, btw.

Christian
  • 191
  • 1
  • 1
  • 8
  • Have you tried different ports, for example 443? – Andrew Domaszek Nov 27 '14 at 14:51
  • Yes and no, I have multiple of these set up and working fine at the moment. This very one worked, as stated, when I tried the connection yesterday over a verified 3G connection. So the setup seems to work as-is under good circumstances. There's no way to change the port used now that I do not have physical access to the unit anymore. – Christian Nov 27 '14 at 14:53
  • Have you tried calling the wireless provider and asking? Have you tried any UDP-based tunneling solutions instead of ssh? Granted you don't have access to the box anymore, which might make things difficult to set up. I would check your packet loss over gprs by logging pings, if you can do any remote execution and disk changes to the embedded unit. That'll give you information about how reliable the network connection for that unit is. – Andrew Domaszek Nov 27 '14 at 15:05
  • I am rather assuming the provider (and the 3G router for all the device cares) do some (very) dirty multiple-NAT things on the GPRS connection that are not done on the 3G connection, for the sake of apparent "speed" to the end-user. The router, to my knowledge, only gets a 10.*.*.* private IP address on the WAN interface, so there _must_ be something in between (but then again it did so on the 3G network without a hitch). – Christian Nov 27 '14 at 15:12
  • Oh, and before calling some nondescript callcenter agents on some far, far away country who take my call on behalf of the provider I'd rather have some meaningful input for them ;) – Christian Nov 27 '14 at 15:30
  • Possible duplicate of [SSH remote port forwarding failed](https://serverfault.com/questions/595323/ssh-remote-port-forwarding-failed) – kasperd Aug 15 '18 at 19:13

2 Answers2

3

The Connection timed out during banner exchange error message may indicate network issues. If you see a socket is established using netstat on both the server and the client, there may be a firewall or packet inspection device that is preventing the SSH connection from being established. We have seen this issue on a system that was behind a protected network and the firewall rules were not configured properly. The firewall was also dropping any connection that tried to initiate an HTTP/1.1 request, which was confirmed using netcat to simulate a web server and connect to it using netcat. When we changed the response to HTTP/1.2 it let the response thru, indicating something in the network layer was inspecting requests and applying filtering rules against them.

Other low level networking issues like MTU/Jumbo frames issues or high packet loss could also cause the problem.

Greg Bray
  • 5,610
  • 5
  • 36
  • 53
  • Yes, I assumed that already. The question is, is there a) a way within ssh to apparently more reliably detect this and restart the connection from the outside client or b) kill one specific, not working ssh connection from the server (eg. inbound) side... – Christian Nov 10 '17 at 09:04
-3

If your firewall (e.g. on a jump host) is using knockd simply try to increase Cmd_Timeout: https://linux.die.net/man/1/knockd

Timm
  • 41
  • 1
  • I am not using `knockd` or a firewall on my machine. Even if the emote side is behind a firewall, my hope was that some actually working timeout mechanism simply restarts the connection... – Christian Sep 21 '18 at 08:16