0

Symptom: The first attempt at connecting to a TCP service (https, ftp, or ssh) fails. The second attempt succeeds Subsequent connections will work until there is a 15 minute period of inactivity, in which case the next connection will fail again. Upon inspection of packets on both sides, the client seems to log a TCP 3 way handshake for the first connection followed by a reset. The server logs no packets. Strangely, this behavior is the same even if the client connects to a port the server is not listening on (for this example I used port 234). Subsequent connections to the non-existent service on port 234 are logged as SYN/Resets, and traffic to supported ports is logged as expected.

Command used to connect to port 234:

$ nc server-ip 234

executed three times

Server is running Ubuntu 20.04.6 LTS and hosted by GoDaddy

Packet dump on client:

1. 11:00:26.963058 IP client-ip.65228 > server-ip.234: Flags [S], seq 4286801981, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 866207816 ecr 0,sackOK,eol], length 0
2. 11:00:27.012703 IP server-ip.234 > client-ip.65228: Flags [S.], seq 1144885311, ack 4286801982, win 16384, options [mss 1460], length 0
3. 11:00:27.012755 IP client-ip.65228 > server-ip.234: Flags [.], ack 1, win 65535, length 0
4. 11:00:27.055571 IP server-ip.234 > client-ip.65228: Flags [R.], seq 1, ack 1, win 16384, length 0
5. 11:00:29.631388 IP client-ip.65229 > server-ip.234: Flags [S], seq 255589775, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 145472704 ecr 0,sackOK,eol], length 0
6. 11:00:29.675475 IP server-ip.234 > client-ip.65229: Flags [R.], seq 0, ack 255589776, win 0, length 0
7. 11:00:32.484897 IP client-ip.65230 >server-ip.234: Flags [S], seq 1572449604, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1092910203 ecr 0,sackOK,eol], length 0
8. 11:00:32.528987 IP server-ip.234 > client-ip.65230: Flags [R.], seq 0, ack 1572449605, win 0, length 0

Packet dump on server (note a 3 hour time difference):

1. no corresponding packets
2. no corresponding packets
3. no corresponding packets
4. no corresponding packets
5. 08:00:29.712757 IP client-ip.65229 > server-ip.234: Flags [S], seq 255589775, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 145472704 ecr 0,sackOK,eol], length 0
6. 08:00:29.712785 IP server-ip.234 > client-ip.65229: Flags [R.], seq 0, ack 255589776, win 0, length 0
7. 08:00:32.566058 IP client-ip.65230 > server-ip.234: Flags [S], seq 1572449604, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1092910203 ecr 0,sackOK,eol], length 0
8. 08:00:32.566085 IP server-ip.234 > client-ip.65230: Flags [R.], seq 0, ack 1572449605, win 0, length 0

Any ideas?

The local firewall is configured like so:

# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

Here is a partial traceroute to the server:

 4  be-50-rar01.kokomo.in.indiana.comcast.net (96.108.120.145)  18.098 ms  18.663 ms  21.537 ms
 5  24.153.88.85 (24.153.88.85)  26.530 ms  23.513 ms  23.843 ms
 6  * 4.68.110.122 (4.68.110.122)  34.668 ms  24.746 ms
 7  * * *
 8  4.14.98.38 (4.14.98.38)  50.381 ms  46.376 ms  43.220 ms
 9  * * *
10  148.72.36.47 (148.72.36.47)  49.768 ms  42.868 ms  44.426 ms
11  148.72.36.59 (148.72.36.59)  42.996 ms  42.648 ms  41.981 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  225.220.167.72.host.secureserver.net (72.167.220.225)  78.721 ms  43.946 ms  41.804 ms
Randall Embry
  • 1
  • 3
  • 1
    We kind of need to know where these two machines are located, what ISPs they are connected to. Clearly there's some intermediate hardware that is fielding the first connection attempt, but you've given us nothing to work on as to where that hardware might be. – tsc_chazz Aug 13 '23 at 16:16
  • Clients are worldwide. It's not isolated to just one place. Server is hosted at GoDaddy. – Randall Embry Aug 13 '23 at 16:21
  • Here is part of the traceroute: 4 be-50-rar01.kokomo.in.indiana.comcast.net (96.108.120.145) 18.098 ms 18.663 ms 21.537 ms 5 24.153.88.85 (24.153.88.85) 26.530 ms 23.513 ms 23.843 ms 6 * 4.68.110.122 (4.68.110.122) 34.668 ms 24.746 ms 7 * * * 8 4.14.98.38 (4.14.98.38) 50.381 ms 46.376 ms 43.220 ms 9 * * * 10 148.72.36.47 (148.72.36.47) 49.768 ms 42.868 ms 44.426 ms 11 148.72.36.59 (148.72.36.59) 42.996 ms 42.648 ms 41.981 ms – Randall Embry Aug 13 '23 at 16:24
  • 1
    Might want to put that traceroute in the question - putting the trace in a code block {} will make it more readable, much more than it appears here. 148.72.36.59 is your server? – tsc_chazz Aug 13 '23 at 16:36
  • Will do. 72.167.220.225 is the server. Run "nc 72.167.220.225 21" twice to see the behavior. 15 minutes later it will do it again. – Randall Embry Aug 13 '23 at 16:46

1 Answers1

0

GoDaddy support says this is some unexplained bug in their hypervisor and they are migrating people away from these servers as a long term solution.

Randall Embry
  • 1
  • 3