1

We have server in AWS that's trying to connect over TCP to an application in an on premise server but fails. Initially we thought it might be routing/firewall issues so we tried to connect to that application using telnet on the specific port it's running while running tcpdump on the on-premise server.

What we (think) see is that we get the telenet connection request but nothing is sent back:

On the AWS server we run: telnet ON_PREMISE_IP PORT

On the On-Premise server we run tcpdump -i INTERFACE -n port PORT

And we see:

15:02:49.785795 IP AWS_IP.RANDOM_PORT > ON_PREMISE_IP.int-rcv-cntrl: Flags [S], seq 3124225083, win 62727, options [mss 1350,sackOK,TS val 576158266 ecr 0,nop,wscale 7], length 0

And on the AWS side we get: telnet: Unable to connect to remote host: No route to host

I've verified that we're listening on that port by running netstat:

sudo netstat -ltnp | grep PORT
tcp        0      0 ON_PREMISE_IP:PORT    0.0.0.0:*               LISTEN      PID/java

In addition to that we're able to establish connection from the on premise server to that same AWS server.

The on premise server is running CentOS 8 with kernel 4.18.0-305.19.1.el8_4.x86_64

I've no idea how to continue from here :/

Update-1: I've used netcat to listen to that port instead of our own Java app but the issue persist.

Tomer Arazy
  • 111
  • 3
  • 1
    Is `PORT` actually 3603? If not, your `tcpdump` data is irrelevant – roaima Jun 21 '22 at 12:50
  • Do you see anything else from `tcpdump`, specifically the `SYN/ACK` that should be returned to the AWS client? – roaima Jun 21 '22 at 12:50
  • The error message on the client doesn't match the situation. `No route to host` means that the client doesn't know how to get to the server at all - not that the server refuses to respond. – roaima Jun 21 '22 at 12:52
  • Yes, it's 3603,how did you know? :-) I don't see anything else in the `tcpdump` – Tomer Arazy Jun 21 '22 at 14:21
  • The fact that your on-premise server is not replying with a SYN-ACK to the SYN that's received, and you do have a process listening on 3603 (int-rcv-ctrl is a standard port in `/etc/services`, btw) suggests a firewall issue on the on-premise server. NB: It's not unheard of for an overly clever admin to setup firewall rules to reply with ICMP messages such as "No route to host" to blocked connections in an attempt to feel more secure (whether that is effective or not is debatable) – Brandon Xavier Jun 22 '22 at 14:34
  • Do you have any software firewall ( probably iptables) running on the on premise server? Also have you looked at the routing on the on-premise server? – eranga Jun 22 '22 at 15:53
  • @BrandonXavier - I'm not aware of such firewall but it's possible, what do I need to check? I'm not very knowledgeable when it comes to networking – Tomer Arazy Jun 22 '22 at 16:37
  • @eranga - The `iptables` seems empty (no specific configuration was added, everything is set to `ACCEPT`) – Tomer Arazy Jun 23 '22 at 06:33
  • Can you try tcpdump for the host like below. tcpdump -i INTERFACE -n host AWS_IP – eranga Jun 23 '22 at 16:12

0 Answers0