We can't connect to a specific external server from inside our LAN.
However we can connect to the same server from outside our LAN, at home, or using a mobile phone. It was working just fine for months, and one day it stopped working. I can't figure out what, if anything, changed when it stopped working.
I'm trying to develop a procedure to troubleshoot this and pinpoint the problem.
The server is a hosted CentOS vps. If I ping any of it's 3 IP addresses from inside our LAN, the ping times out. If I try to connect to web servers on the vps, it times out. If I perform a tracert, I get these results:
Tracing route to static-161-150-73-69.nocdirect.com [69.73.150.161] over a maximum of 30 hops:
1 * * * Request timed out.
2 * * * Request timed out.
3 34 ms 10 ms 30 ms 96.120.4.229
4 10 ms 12 ms 8 ms xe-11-1-0-32767-sur01.n4atlanta.ga.atlanta.comcast.net [68.85.68.61]
5 9 ms 15 ms 12 ms xe-9-0-2-0-ar01.b0atlanta.ga.atlanta.comcast.net [68.86.106.182]
6 14 ms 23 ms 23 ms he-5-12-0-0-10-cr01.56marietta.ga.ibone.comcast.net [68.86.93.125]
7 16 ms 15 ms 15 ms pos-0-1-0-0-pe01.56marietta.ga.ibone.comcast.net [68.86.86.86]
8 21 ms 12 ms 16 ms 173.167.57.134
9 * 19 ms 18 ms ae3-20g.cr2.atl1.us.nlayer.net [69.31.135.165]
10 18 ms 13 ms 16 ms as46562.ae6-1677.cr2.atl1.us.nlayer.net [198.47.120.146]
11 11 ms 16 ms 25 ms 184.170.245.254
12 13 ms 21 ms 15 ms 209.140.17.58
13 23 ms 12 ms 19 ms arya.nocdirect.com [69.73.150.66]
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * [10.0.0.144] reports: Destination host unreachable.
Trace complete.
Our LAN is behind a D-link DFL-800 firewall. I can't see any rules on the firewall to block traffic to these 3 addresses.
I CAN ping other IP addresses that are probably at the hosting company, IP's that are slightly above and below the IP range.
Here's a successful tracert from outside our LAN
Tracing route to static-161-150-73-69.nocdirect.com [69.73.150.161] over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 192.168.1.1
2 28 ms 22 ms 26 ms c-76-111-52-1.hsd1.ga.comcast.net [76.111.52.1]
3 9 ms 8 ms 10 ms xe-11-0-0-32767-sur01.b6powsprings.ga.atlanta.comcast.net [68.85.68.109]
4 53 ms 17 ms 13 ms xe-11-1-1-0-ar01.b0atlanta.ga.atlanta.comcast.net [68.85.108.169]
5 19 ms 23 ms 23 ms he-5-13-0-0-10-cr01.56marietta.ga.ibone.comcast.net [68.86.93.201]
6 12 ms 15 ms 12 ms pos-0-11-0-0-pe01.56marietta.ga.ibone.comcast.net [68.86.88.186]
7 13 ms 16 ms 18 ms 173.167.57.134
8 11 ms 17 ms 10 ms ae0-50g.cr1.atl1.us.nlayer.net [69.31.135.129]
9 13 ms 13 ms 10 ms ae1-40g.cr2.atl1.us.nlayer.net [69.31.135.138]
10 12 ms 12 ms 11 ms as46562.ae6-1677.cr2.atl1.us.nlayer.net [198.47.120.146]
11 18 ms 70 ms 23 ms 184.170.245.254
12 12 ms 16 ms 16 ms 209.140.17.58
13 13 ms 13 ms 11 ms arya.nocdirect.com [69.73.150.66]
14 15 ms 20 ms 13 ms static-161-150-73-69.nocdirect.com [69.73.150.161]
Trace complete.
Resolution:
I SSH'd into the server and followed the procedure outlined in the following blog post.
How to Remove and Add Rules to IP Tables Chains in Centos Linux
1 Dump all the rules to a file
iptables -L -n --line-numbers > /tmp/ip.tables
2 Edit the file using vi
vi /tmp/ip.tables
3 Locate LOCALINPUT or LOCALOUTPUT rules that DROP your IP. Make a note of the rule number.
4 Delete the rules
iptables -D LOCALINPUT {rule#}
iptables -D LOCALOUTPUT {rule#}
5 Add rules to allow your IP
iptables -A INPUT -p all -s {IP} -j ACCEPT
iptables -A OUTPUT -p all -s {IP} -j ACCEPT
