0

We have 17k tcp connection in CLOSING state on redis server. My understanding is that only client would initiate closing a connection, not from a redis server. So I should see 0 connection in CLOSING state on redis server. Does anyone have some experience and know the root cause?

The actual connection# in 'redis-cli info' is less than 200 and we have more than enough the memory and cpu idle time is more than 90%.

I went through redis doc on http://redis.io/topics/clients. None of the items there could help me finding the root cause.

redis version: 2.6.7

netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
  1 CLOSE_WAIT
  16980 CLOSING
  128 ESTABLISHED
  21 LAST_ACK
  12 LISTEN
  1 SYN_RECV
Chuan Ma
  • 131
  • 1
  • 4
  • 1
    The connection is waiting for an `ACK`. Can you look at the network traffic with `wireshark` or `tcpdump`? – ott-- Mar 16 '15 at 16:31
  • Yeah. It's waiting for an `ACK`. But my question was that it should be client initiating the closing connection flow, not the redis server. http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html#fn:serverfirst. I ran tcpdump and didn't find anything suspicious yet. – Chuan Ma Mar 16 '15 at 18:32
  • You're not showing your log? Maybe others or me detect something suspicious? – ott-- Mar 16 '15 at 20:42
  • thanks for your help. Using tcpdump and wireshark leads me to the right direction. I am not allowed to post the log data here. So I had to investigate the log myself. In the end, I found out that's a bug on our phpredis client library. – Chuan Ma Mar 23 '15 at 14:36

1 Answers1

1

This is caused by the phpredis library we are using to connect to the redis server.

Basically phpredis sends a QUIT command to ask the redis server to close the connection. But right after doing that, phpredis closes the tcp socket itself, causing both sides trying to close the connection. Therefore the server has so many connections stuck at CLOSING state.

I've created a simple fix for that issue. https://github.com/phpredis/phpredis/issues/562

Chuan Ma
  • 131
  • 1
  • 4