1

I have a Centos 7 box I inherited where if I try to connect to an unused localhost port, the connection fails with a timeout instead of being immediately refused. Example below:

$ nc 127.0.0.1 5800
Ncat: Connection timed out.

There is definitely nothing running on the port:

$ netstat -tna | grep 5800
$

tcpdump shows the following:

$ sudo tcpdump -v -i lo port 5800                                                                                                                                                         
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
13:46:02.547857 IP (tos 0x0, ttl 64, id 52160, offset 0, flags [DF], proto TCP (6), length 60)
    localhost.37184 > localhost.5800: Flags [S], cksum 0xfe30 (incorrect -> 0xe3e7), seq 3475068921, win 65535, options [mss 65495,sackOK,TS val 11443257 ecr 0,nop,wscale 14], length 0
13:46:03.550503 IP (tos 0x0, ttl 64, id 52161, offset 0, flags [DF], proto TCP (6), length 60)
    localhost.37184 > localhost.5800: Flags [S], cksum 0xfe30 (incorrect -> 0xdffc), seq 3475068921, win 65535, options [mss 65495,sackOK,TS val 11444260 ecr 0,nop,wscale 14], length 0
13:46:05.554562 IP (tos 0x0, ttl 64, id 52162, offset 0, flags [DF], proto TCP (6), length 60)
    localhost.37184 > localhost.5800: Flags [S], cksum 0xfe30 (incorrect -> 0xd828), seq 3475068921, win 65535, options [mss 65495,sackOK,TS val 11446264 ecr 0,nop,wscale 14], length 0
13:46:09.562520 IP (tos 0x0, ttl 64, id 52163, offset 0, flags [DF], proto TCP (6), length 60)
    localhost.37184 > localhost.5800: Flags [S], cksum 0xfe30 (incorrect -> 0xc880), seq 3475068921, win 65535, options [mss 65495,sackOK,TS val 11450272 ecr 0,nop,wscale 14], length 0

/etc/sysconfig/network-scripts/ifcfg.lo looks normal:

DEVICE=lo
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback

Disabling the firewall has no effect. On another Centos 7 box that I had set up, it works as expected.

$ nc 127.0.0.1 5800
Ncat: Connection refused.

I haven't been able to find any significant configuration differences between the inherited box and the one I set up. I really don't want to re-image the inherited box if I don't have to.

What else should I be looking at to fix this? I have an application that takes way too long to initialize due to the timeouts.

EDIT:

Here's the result of the iptables -L -n command:

$ sudo iptables -n -L -v
Chain INPUT (policy ACCEPT 249K packets, 26M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  virbr0 *       192.168.122.0/24     0.0.0.0/0           
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0           
    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT 248K packets, 26M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:68

$ sudo iptables -L -n                                                                                                                                                                   
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
INPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0           
INPUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0           
INPUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
FORWARD_direct  all  --  0.0.0.0/0            0.0.0.0/0           
FORWARD_IN_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0           
FORWARD_IN_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
FORWARD_OUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0           
FORWARD_OUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
OUTPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0           

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination         
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination         
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_direct (1 references)
target     prot opt source               destination         

Chain FWDI_public (2 references)
target     prot opt source               destination         
FWDI_public_log  all  --  0.0.0.0/0            0.0.0.0/0           
FWDI_public_deny  all  --  0.0.0.0/0            0.0.0.0/0           
FWDI_public_allow  all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           

Chain FWDI_public_allow (1 references)
target     prot opt source               destination         

Chain FWDI_public_deny (1 references)
target     prot opt source               destination         

Chain FWDI_public_log (1 references)
target     prot opt source               destination         

Chain FWDO_public (2 references)
target     prot opt source               destination         
FWDO_public_log  all  --  0.0.0.0/0            0.0.0.0/0           
FWDO_public_deny  all  --  0.0.0.0/0            0.0.0.0/0           
FWDO_public_allow  all  --  0.0.0.0/0            0.0.0.0/0           

Chain FWDO_public_allow (1 references)
target     prot opt source               destination         

Chain FWDO_public_deny (1 references)
target     prot opt source               destination         

Chain FWDO_public_log (1 references)
target     prot opt source               destination         

Chain INPUT_ZONES (1 references)
target     prot opt source               destination         
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain INPUT_direct (1 references)
target     prot opt source               destination         

Chain IN_public (2 references)
target     prot opt source               destination         
IN_public_log  all  --  0.0.0.0/0            0.0.0.0/0           
IN_public_deny  all  --  0.0.0.0/0            0.0.0.0/0           
IN_public_allow  all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           

Chain IN_public_allow (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW,UNTRACKED

Chain IN_public_deny (1 references)
target     prot opt source               destination         

Chain IN_public_log (1 references)
target     prot opt source               destination         

Chain OUTPUT_direct (1 references)
target     prot opt source               destination

and the output of ip addr list:

$ sudo ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 40:16:7e:a5:10:b2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.10/24 brd 192.168.100.255 scope global noprefixroute enp4s0
       valid_lft forever preferred_lft forever
    inet6 fe80::de71:884c:6887:9205/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether b0:6e:bf:c6:0e:09 brd ff:ff:ff:ff:ff:ff
4: wlp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 4000
    link/ether 9e:75:ce:d2:2e:ae brd ff:ff:ff:ff:ff:ff
5: wlp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 02:a2:db:06:60:91 brd ff:ff:ff:ff:ff:ff
6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:ca:11:07 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
7: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:ca:11:07 brd ff:ff:ff:ff:ff:ff

output of iptables -t nat -n -L:

$ sudo iptables -t nat -n -L -v
Chain PREROUTING (policy ACCEPT 42 packets, 11019 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 42 packets, 11019 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 7 packets, 1850 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 7 packets, 1850 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  *      *       192.168.122.0/24     224.0.0.0/24        
    0     0 RETURN     all  --  *      *       192.168.122.0/24     255.255.255.255     
    0     0 MASQUERADE  tcp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
    0     0 MASQUERADE  udp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
    0     0 MASQUERADE  all  --  *      *       192.168.122.0/24    !192.168.122.0/24    

$ sudo iptables -t nat -n -L                                                                                                                                      
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
PREROUTING_direct  all  --  0.0.0.0/0            0.0.0.0/0           
PREROUTING_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0           
PREROUTING_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
OUTPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
POSTROUTING_direct  all  --  0.0.0.0/0            0.0.0.0/0           
POSTROUTING_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0           
POSTROUTING_ZONES  all  --  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT_direct (1 references)
target     prot opt source               destination         

Chain POSTROUTING_ZONES (1 references)
target     prot opt source               destination         
POST_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
POST_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 

Chain POSTROUTING_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain POSTROUTING_direct (1 references)
target     prot opt source               destination         

Chain POST_public (2 references)
target     prot opt source               destination         
POST_public_log  all  --  0.0.0.0/0            0.0.0.0/0           
POST_public_deny  all  --  0.0.0.0/0            0.0.0.0/0           
POST_public_allow  all  --  0.0.0.0/0            0.0.0.0/0           

Chain POST_public_allow (1 references)
target     prot opt source               destination         

Chain POST_public_deny (1 references)
target     prot opt source               destination         

Chain POST_public_log (1 references)
target     prot opt source               destination         

Chain PREROUTING_ZONES (1 references)
target     prot opt source               destination         
PRE_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 
PRE_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto] 

Chain PREROUTING_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain PREROUTING_direct (1 references)
target     prot opt source               destination         

Chain PRE_public (2 references)
target     prot opt source               destination         
PRE_public_log  all  --  0.0.0.0/0            0.0.0.0/0           
PRE_public_deny  all  --  0.0.0.0/0            0.0.0.0/0           
PRE_public_allow  all  --  0.0.0.0/0            0.0.0.0/0           

Chain PRE_public_allow (1 references)
target     prot opt source               destination         

Chain PRE_public_deny (1 references)
target     prot opt source               destination         

Chain PRE_public_log (1 references)
target     prot opt source               destination
pcarter
  • 107
  • 3
  • Could you still post the output of for example `/sbin/iptables -L -n` on the troublesome box? – Janne Paalijarvi Aug 03 '20 at 18:17
  • @JannePaalijarvi Added the output as requested. – pcarter Aug 03 '20 at 19:02
  • Thanks. Can you also show `ip addr list` ? – Janne Paalijarvi Aug 03 '20 at 20:19
  • @JannePaalijarvi Added – pcarter Aug 03 '20 at 22:37
  • Thanks again. I'm sorry I don't yet have a solution. But I have couple of new things that could be helpful for me or other readers. Post output of `/sbin/iptables -t nat -n -L` Also, what happens if you run a listening `nc` on the specific port and try to connect to it? Also, what happens when connecting to another port than 5800? – Janne Paalijarvi Aug 04 '20 at 07:31
  • @JannePaalijarvi Thanks for looking at this. Added the new output. There's nothing special about port 5800. The same happens on every inactive port I try. Using `nc` to listen and connect works as expected. Also, stopping `firewalld` has no effect. Doesn't that rule out the firewall settings? – pcarter Aug 04 '20 at 11:22
  • The firewall listings are incomplete. You also need to add `-v` to the commands. – Michael Hampton Aug 04 '20 at 11:23
  • @MichaelHampton Which command? – pcarter Aug 04 '20 at 11:25
  • The `iptables` commands, of course! What other command shows the firewall listings? – Michael Hampton Aug 04 '20 at 11:27
  • @MichaelHampton OK, I added them, but I'm still confused how the firewall settings could be the problem, if it persists with the firewall off. – pcarter Aug 04 '20 at 11:40
  • It might not be relevant. The point is to rule things out with some certainty. And it needs to work with the firewall on, anyway. Also it looks like you posted the iptables output with firewalld off. – Michael Hampton Aug 04 '20 at 11:49
  • @MichaelHampton Yeah, I just noticed that too and fixed it. – pcarter Aug 04 '20 at 11:52
  • A firewall can still be active outside of your machine. And a timeout usually points toward a firewall. – Gerald Schneider Aug 04 '20 at 11:58
  • @GeraldSchneider, I'm seeing this problem on the localhost loopback interface. – pcarter Aug 04 '20 at 12:04

0 Answers0