From past 2 days, I'm struggling with a weird issue, found no answers related to my problem, earlier everything was working fine, but suddenly it stopped.
Issue exists only on my client system, from any other system everything works fine. I'm trying to use a proxy say 201.17.10.12:11111 for curl request and it fails with "Connection timed out". wget, ssh also fails from client server. nmap & ping works.
I checked with netstat -nputw after curl request and it shows connection in SYN_SENT state, also analyzed with tcpdump, results are
$ tcpdump host 201.17.10.12 -w packets &
$ curl -x http://201.17.10.12:11111 http://www.example.com
curl: (7) Failed to connect to 201.17.10.12 port 11111: Connection timed out
$ tcpdump -nn -r packets | less
22:04:00.556502 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2550685 ecr 0,nop,wscale 7], length 0
22:04:01.556560 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2550936 ecr 0,nop,wscale 7], length 0
22:04:03.560552 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2551437 ecr 0,nop,wscale 7], length 0
22:04:07.572559 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2552440 ecr 0,nop,wscale 7], length 0
22:04:15.588592 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2554444 ecr 0,nop,wscale 7], length 0
22:04:31.604579 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2558448 ecr 0,nop,wscale 7], length 0
22:05:03.668547 IP 10.2.2.57.62544 > 201.17.10.12.11111: Flags [S], seq 2069349153, win 29200, options [mss 1460,sackOK,TS val 2566464 ecr 0,nop,wscale 7], length 0
~
But now if check with nmap, it shows a response
$ nmap -sS -Pn -p 11111 -T4 -vv --reason -oN nmap.results 201.17.10.12
Starting Nmap 6.40 ( http://nmap.org ) at 2017-05-18 22:03 IST
Initiating Parallel DNS resolution of 1 host. at 22:03
Completed Parallel DNS resolution of 1 host. at 22:03, 0.41s elapsed
Initiating SYN Stealth Scan at 22:03
Scanning 201.17.10.12 (201.17.10.12) [1 port]
Discovered open port 11111/tcp on 201.17.10.12
Completed SYN Stealth Scan at 22:03, 0.21s elapsed (1 total ports)
Nmap scan report for 201.17.10.12 (201.17.10.12)
Host is up, received user-set (0.16s latency).
Scanned at 2017-05-18 22:03:23 IST for 0s
PORT STATE SERVICE REASON
11111/tcp open unknown syn-ack
Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.68 seconds
Raw packets sent: 1 (44B) | Rcvd: 1 (44B)
Firewall & apparmor is not running on both client and proxy server (Ubuntu 14.04). Please help since I'm unable to think now, if nmap/ping are showing that everything is fine then why it is failing to connect when using as proxy and what could i do to resolve this issue?
EDIT 1 curl with -vv shows:
$ curl -vv -x http://201.17.10.12:11111 http://www.google.com
* Rebuilt URL to: http://www.google.com/
* Hostname was NOT found in DNS cache
* Trying 201.17.10.12...
* connect to 201.17.10.12 port 11111 failed: Connection timed out
* Failed to connect to 201.17.10.12 port 11111: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to 201.17.10.12 port 11111: Connection timed out
EDIT 2:
I just noticed this is happening with any program i use on client, ssh/wget/curl. Only nmap and ping works. If i try to connect to server to any non-open ports, it immediately exits with message
$ ssh root@201.17.10.12 -p 32
ssh: connect to host 201.17.10.12 port 32: Connection refused
but if try to connect to any open/listening ports it waits till "Connection timed out" Checking port 32 with nmap shows State:Closed, Reason:Reset. and checking open port 22 with nmap shows State:Open, Reason:syn-ack. Means nmap is getting syn-ack response properly, whether other programs are either not receiving response or not able to read response and waits till timeout occurs.
Output of iptables -L -v -n at server
$ iptables -L -v -n
Chain INPUT (policy ACCEPT 2334K packets, 1925M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 2605K packets, 778M bytes)
pkts bytes target prot opt in out source destination
Output of iptables -L -v -n at client
$ iptables -L -v -n
Chain INPUT (policy ACCEPT 7219K packets, 1845M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 8616K packets, 1288M bytes)
pkts bytes target prot opt in out source destination
