4

A quite busy proxy server has lots of "SYNs to LISTEN sockets dropped".

I learned one cause could be a too small backlog size. But in that case the "times the listen queue of a socket overflowed" value should be equal (which it is not).

So what could be a cause for this behaviour? Maybe a broken nic?

We have 5 proxies, in 2 of which the two numbers are not equal, so this problem seems to be happening there.

Here the output from netstat:

$ netstat -s | grep -i list
238627 times the listen queue of a socket overflowed
8610307 SYNs to LISTEN sockets dropped

the servers have ipv4 and ipv6 traffic, maybe that helps?

edlerd
  • 826
  • 8
  • 13

2 Answers2

7

These counters ultimately come from the kernel and map to the LINUX_MIB_LISTENOVERFLOWS and LINUX_MIB_LISTENDROPS counters. You can see from the source of net/ipv4/tcp_ipv4.c(tcp_v4_syn_recv_sock) around line #1392 that when LINUX_MIB_LISTENOVERFLOWS is incremented, LINUX_MIB_LISTENDROPS will also be incremented but there are exit conditions where only the latter can be incremented so it's not a bug that they don't match.

In the same file you can see there's this code:

1291 int tcp_v4_conn_request(struct sock *sk, struct sk_buff *skb)
1292 {
1293         /* Never answer to SYNs send to broadcast or multicast */
1294         if (skb_rtable(skb)->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST))
1295                 goto drop;
1296 
1297         return tcp_conn_request(&tcp_request_sock_ops,
1298                                 &tcp_request_sock_ipv4_ops, sk, skb);
1299 
1300 drop:
1301         NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_LISTENDROPS);
1302         return 0;
1303 }

So you can see at least one cause is a SYN to a broadcast or multicast address.

bodgit
  • 4,751
  • 16
  • 27
  • Thanks for the explanation. From the code i see two possibilities for growing LINUX_MIB_LISTENOVERFLOWS: 1) syn to broadcast as mentioned by you 2) Still in SYN_RECV, just remove it silently. (i honestly dont understand what is meant there). As the counts diverge on 2 of 5 proxies (which are all configured in the same way), can you suggest a cause for the behaviour? – edlerd Nov 24 '14 at 12:25
  • Nothing definitive, however I would monitor the value of both counters over time and see what the general rate and pattern is. Bear in mind there is a separate yet similar code path for TCP & IPv6 so you might have non-unicast IPv6 traffic causing the counters to increase. – bodgit Nov 24 '14 at 13:32
  • i was monitoring the numbers again today. so the result is, that the SYNs to LISTEN sockets dropped is increased by ~100k, while the other counter is unchanged. Also I can see the first value change every couple of seconds. Any idea how to identify the reason? – edlerd Nov 26 '14 at 21:54
2

Usually wmem and rmem defaults are 212992 bytes. Apparently not enough on busy server. Raised to 8MB and the problem disappeared.

sysctl -w net.core.wmem_default=8388608
sysctl -w net.core.rmem_default=8388608
Arie Skliarouk
  • 608
  • 1
  • 6
  • 12
  • If you are already setting net.ipv4.tcp_rmem and net.ipv4.tcp_wmem and you are only serving TCP traffic, is there a point in also setting the net core values? – andresp May 13 '19 at 10:44
  • According to IBM those net.core properties are superseeded by the TCP ones for TCP connections: https://www.ibm.com/support/knowledgecenter/en/linuxonibm/liaag/wkvm/wkvm_c_tune_tcpip.htm so there is no point in setting them if you already set the ones for TCP. – andresp May 13 '19 at 10:46