1

We have a strange problem. Our remote server shows over 900 connections coming from our office over port 80:

# ss -n | grep ESTAB | grep $OFFICEIP | grep :80 | wc -l
935

Netstat shows a similar number:

# netstat -n | grep $OFFICEIP | grep :80 | wc -l
922

(Although interestingly the netstat number periodically changes, either going up or down a bit.)

We simply cannot find where these connections are coming from. Our office router doesn't show them in its tables, and doing a tcpdump (tcpdump -ieth1 -qnnvvSXs 1500 port 80) on the remote host shows zero traffic on that port at all, let alone from our office ip.

In addition, conntrack doesn't show the connections as existing either:

# conntrack -L | grep $OFFICEIP | grep 80
conntrack v0.9.13 (conntrack-tools): 38025 flow entries have been shown.

This is happening on a box running CentOS 6, kernel version 2.6.32-431.29.2.el6.x86_64. If anyone has any ideas as to where these discrepancies are coming from it would be greatly appreciated.

Mediocre Gopher
  • 803
  • 1
  • 13
  • 24
  • 1
    Why don't you just remove the `| wc -l` to get the whole list and check the other side's IP? Also, I would recommend using `grep -F $OFFICEIP` instead, since the dot in grep (used without the `-F` flag) means "any character" and could be getting wrong results. – NuTTyX Jan 15 '15 at 20:05
  • Our office is NAT'd, so the other side is always `$OFFICEIP`. I tried with the `-F`, it didn't make a difference though. – Mediocre Gopher Jan 15 '15 at 20:06
  • If the connections are over port 80, did the web server get any logs from it? `netstat -nap | grep -F $OFFICEIP | grep :80` should give you the process that is attending that connection, so you could either restart it (to force the connections to close) or check its logs. – NuTTyX Jan 15 '15 at 20:12
  • Unfortunately the "connections" (or whatever they are) are already made, and I didn't have logging on back then so it's too late for that. And I can't restart it at the moment, I might try that later tonight though. – Mediocre Gopher Jan 15 '15 at 20:47
  • i have a similar problem, but more isolated. 2 rhel machines talking over a switch. one machine believes that it has M connection ESTABLISHED to the other, and that other lists N connections to the first machine. M does not equals N. This happened during stress-test during which the server sent [FIN], but the packets get lost sometimes, and if [FIN] didn't reach the client, server quickly kills FIN_WAIT state. This is what I believe happened. Default keepalive settings keep broken connections for hours... – wick Feb 03 '15 at 21:44

0 Answers0