6

I'm running a memcached server which is serving two front end servers. These are connecting to the memcached server from Apache2/PHP using the standard PECL library.

For some reason the memcached on the server is consuming more and more CPU until it reaches 100%.

I'm runnign memcached with the following parameters:

/usr/bin/memcached -d -m 128 -p 11211 -u memcache -l 0.0.0.0

Another interesting thing I've found out is that there are a lot's of open connections (200) between the front end servers and the memcached server

netstat -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0     30 10.2.10.69:11211        10.2.10.193:49393       ESTABLISHED
tcp        0     30 10.2.10.69:11211        10.2.10.172:52918       ESTABLISHED
tcp        0     30 10.2.10.69:11211        10.2.10.193:39677       ESTABLISHED
tcp        0     30 10.2.10.69:11211        10.2.10.193:46480       ESTABLISHED
tcp        0     30 10.2.10.69:11211        10.2.10.172:58869       ESTABLISHED
tcp        0     30 10.2.10.69:11211        10.2.10.193:38498       ESTABLISHED
tcp        0      0 10.2.10.69:11211        10.2.10.193:54095       ESTABLISHED
tcp        0     30 10.2.10.69:11211        10.2.10.172:55895       ESTABLISHED
tcp        0     30 10.2.10.69:11211        10.2.10.193:38809       ESTABLISHED
...

And the most of them has the value "30" in the "Send-Q" column. These connections are present in "ESTABLISHED" mode for several minutes until they disappear from the output of netstat. Does this mean anything?

Also my Memcached "stats":

STAT pid 1183
STAT uptime 5087
STAT time 1398177110
STAT version 1.4.13
STAT libevent 2.0.16-stable
STAT pointer_size 64
STAT rusage_user 322.404149
STAT rusage_system 1055.877988
STAT curr_connections 164
STAT total_connections 28962
STAT connection_structures 291
STAT reserved_fds 20
STAT cmd_get 55986225
STAT cmd_set 22250
STAT cmd_flush 0
STAT cmd_touch 0
STAT get_hits 55891017
STAT get_misses 95208
STAT delete_misses 23066
STAT delete_hits 6277
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 966125003
STAT bytes_written 12255690623
STAT limit_maxbytes 134217728
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT hash_power_level 16
STAT hash_bytes 524288
STAT hash_is_expanding 0
STAT expired_unfetched 0
STAT evicted_unfetched 0
STAT bytes 1898162
STAT curr_items 978
STAT total_items 22250
STAT evictions 0
STAT reclaimed 0
END
filo891
  • 81
  • 1
  • 5
  • Why so little RAM? – Joel E Salas Apr 22 '14 at 14:25
  • you mean the "-m 128" switch? I had there "-m 512" before, but decreased it to check if maybe that is the issue... but it was not. So I just forgot to change it back. – filo891 Apr 22 '14 at 14:27
  • That's still very little RAM – Joel E Salas Apr 22 '14 at 14:28
  • So how much do you suggest to use from 4GB total RAM on a machine which needs approximatelly 1.5GBs for aditional services runnig on it? – filo891 Apr 22 '14 at 14:31
  • So I increased the RAM to 1.5GB and the problem is still present. For some time the connections count was around 10, then it started increasing along with the CPU usage. So it seems like memcached is keeping a queue and it is not able to process it in real time so the queue is just increasing. – filo891 Apr 22 '14 at 15:04

1 Answers1

2

It seems the problem is not with memcached. The connection count was increasing along with the newly created apache processes on the frontend servers and as these are not terminated, they are staying connected to the memcached server.

So I consider this as an issue with my Apache server and it is not connected with this topic anymore.

filo891
  • 81
  • 1
  • 5