0

I have a server using a proxy service (WAF etc) which forwards packets to my server.

I can see established SSL conenctions from all proxy netstat -an and the rest stuck in SYN_RECV:

tcp        0      0 192.168.102.11:443      185.93.230.20:64966     SYN_RECV
tcp        0      0 192.168.102.11:443      192.88.135.20:8306      SYN_RECV
tcp        0      0 192.168.102.11:443      66.248.202.20:10750     SYN_RECV
tcp        0      0 192.168.102.11:443      185.93.230.20:2213      SYN_RECV
tcp        0      0 192.168.102.11:443      66.248.202.20:7494      SYN_RECV
tcp        0      0 192.168.102.11:443      185.93.231.20:32752     ESTABLISHED
tcp        0      0 192.168.102.11:443      185.93.231.20:31910     ESTABLISHED

I can see traffic hit tcpdump port 443 and '(tcp-syn|tcp-ack)!=0' -nn:

For 185.93.231.20.2139

20:36:35.263777 IP 192.168.102.11.443 > 185.93.231.20.2139: Flags [FP.], seq 203642186:203642217, ack 1968471817, win 258, options [nop,nop,TS val 32827456 ecr 876705214], length 31
20:36:36.901357 IP 192.168.102.11.443 > 185.93.231.20.2137: Flags [P.], seq 418165034:418165065, ack 2875697257, win 258, options [nop,nop,TS val 32829093 ecr 876704135], length 31

For 185.93.230.20

20:36:49.098560 IP 185.93.230.20.20721 > 192.168.102.11.443: Flags [S], seq 2855805773, win 29200, options [mss 1460,sackOK,TS val 882921029 ecr 0,nop,wscale 9], length 0
20:36:49.098638 IP 192.168.102.11.443 > 185.93.230.20.20721: Flags [S.], seq 268496949, ack 2855805774, win 28960, options [mss 1460,sackOK,TS val 32841290 ecr 882921029,nop,wscale 7], length 0

For 66.248.202.20:

20:37:02.042048 IP 66.248.202.20.49557 > 192.168.102.11.443: Flags [S], seq 3837436386, win 29200, options [mss 1460,sackOK,TS val 791596242 ecr 0,nop,wscale 9], length 0
20:37:02.042116 IP 192.168.102.11.443 > 66.248.202.20.49557: Flags [S.], seq 2339555392, ack 3837436387, win 28960, options [mss 1460,sackOK,TS val 32854234 ecr 791596242,nop,wscale 7], length 0

For 192.88.135.20:

20:36:39.595087 IP 185.93.228.20.23354 > 192.168.102.11.443: Flags [S], seq 1334433323, win 29200, options [mss 1460,sackOK,TS val 274977072 ecr 0,nop,wscale 9], length 0
20:36:39.595120 IP 192.168.102.11.443 > 185.93.228.20.23354: Flags [S.], seq 1203016390, ack 1334433324, win 28960, options [mss 1460,sackOK,TS val 32831787 ecr 274970056,nop,wscale 7], length

But only traffic from 185.93.231.20 is getting logged in domlogs:

185.93.231.20 - - [22/May/2020:19:55:37 +0400] "GET /blog/video-gallery/ HTTP/1.1" 200 12716 "https://www.example.com/blog/publications/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0" 747893
185.93.231.20 - - [22/May/2020:19:55:39 +0400] "GET /wp-content/uploads/2020/02/Thumbnail72.jpg HTTP/1.1" 200 181941 "https://www.example.com/blog/video-gallery/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0" 1283052
185.93.231.20 - - [22/May/2020:19:55:39 +0400] "GET /wp-content/uploads/2020/02/Thumbnail68.jpg HTTP/1.1" 200 180934 "https://www.example.com/blog/video-gallery/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0" 952373

Any ideas as to what to check next? I have disabled all firewall rules and ensured NAT is correctly working between WAN and host (inbound and out) - no config changes happened this just stopped working.

morleyc
  • 1,150
  • 13
  • 47
  • 89

1 Answers1

0

This turned out to be an asymmetrical routing issue, in that packets could reach the server but the network was not able to return them (due to a routing failure).

netstat with partial connections stuck in SYN_RECV and tcpdump with its flags being returned:

  • [S] inbound from remote server to us to SYN establish a connection
  • [S.] reply from us to remote server to SYN+ACK establish connection request

This was identified on the server with the below SYN request from the remote server:

20:36:49.098560 IP 185.93.230.20.20721 > 192.168.102.11.443: Flags [S], seq 2855805773, win 29200, options [mss 1460,sackOK,TS val 882921029 ecr 0,nop,wscale 9], length 0

Which put the socket into a half-open state:

tcp        0      0 192.168.102.11:443      185.93.230.20:64966     SYN_RECV

And then we respond (note the flags S. meaning SYN+ACK):

20:36:49.098638 IP 192.168.102.11.443 > 185.93.230.20.20721: Flags [S.], seq 268496949, ack 2855805774, win 28960, options [mss 1460,sackOK,TS val 32841290 ecr 882921029,nop,wscale 7], length 0

But this never reaches the remote server, so it never in turn responds with further packets that would complete the handshake and set the socket to ESTABLISHED.

This was resolved by the ISP and associated counter parties to resolve routing issues.

This could have also indicated firewalls dropping packets or misconfigured NAT rules, these were ruled out prior to reaching out to ISP.

morleyc
  • 1,150
  • 13
  • 47
  • 89