5

Im running a website with high traffic peaks (around 10k online users) and Im facing this error with Predis when the site exceeds around 7k online users:

PHP Fatal error:  Uncaught exception 'Predis\\Connection\\ConnectionException' with message 'Connection timed out [tcp://127.0.0.1:6379]'

The server is Intel Xeon 3.4GHz, 32GB RAM and SSD disks, with nginx for static content, apache and mysql. Memory consumption never exceeds 9GB and CPU load is 4 in the peaks.

Im not seeing PHP ouf of memory errors, nor MySQL gone away but only this Redis error. I put timeout 0 and ?read_write_timeout=0 with no luck. Can you help me?

Thanks and sorry for my english!

j0k
  • 22,600
  • 28
  • 79
  • 90
Miguel Angel Ruiz
  • 255
  • 3
  • 9
  • Seeing the same error here on a very similar ubuntu 40GB RAM machine. Our latest working theory is that it's an OS-level issue (hitting some TCP setting limit), but AFAIK we've raised all the relevant settings on both php client and redis server machines (ip_conntrack_tcp_timeout_time_wait=1,nf_conntrack_tcp_timeout_established=600, tcp_tw_recycle=1, tcp_tw_reuse=1, and made nf_conntrack_max and somaxconn sufficiently high). Totally scratching our heads. – Just another hacker Sep 29 '13 at 15:44
  • Im using Ubuntu 12.04 too, and changed to apc store because of this error, but I would like to solve it because Redis gives me some things that apc doesnt. – Miguel Angel Ruiz Sep 29 '13 at 19:09
  • Yeah AFAIK, APC doesn't support the data structures that Redis does. In any case, we went through a whole list of things to debug this, maybe I'll include them here and we can make progress together. Have you checked the file limits descriptor on both machines (for the users that runs the webserver/Redis)? Have you raised the TCP limits I mentioned above? I'm currently testing setting the redis timeout variable to 600 -- maybe the OS is killing the connection underneath Redis and PHP hasn't realized this. Will report back. – Just another hacker Sep 29 '13 at 21:56
  • 1
    Looks like changing redis timeout to 600 worked for us. You can do this by setting the timeout in the /etc/redis/redis.conf file (timeout 600) and then running SET CONFIG timeout 600 from the redis-cli tool (this way you won't need to restart the redis server). – Just another hacker Sep 30 '13 at 03:29

1 Answers1

3

Try setting the Redis "timeout" parameter in /etc/redis/redis.conf to some non-zero value. This almost entirely resolved the issue for us.

Just another hacker
  • 166
  • 1
  • 1
  • 8