3

is it possible to change the error message 'max number of client reached' to null or empty string?

I'm using redis as a cache for my DB values and in cases that I can't get the values from the cache I will get it from the DB. if I could configure it in the redis it self it would be the best option for me because my code won't have to change in order to support that edge case.

if someone has some tips on how to avoid such errors it would be nice as well :) (I'm using php scripts with predis package)

silver
  • 1,633
  • 1
  • 20
  • 32
  • IMO that's better to handle in client. if you wants to do that, you can maybe set that value to an unreasonably high value ? – Tuan Anh Tran Oct 12 '20 at 08:51
  • I have multiple clients and I don't want to change all of them... currently it is set to 10K how much do you think is possible? – silver Oct 12 '20 at 09:06
  • 10k is plenty already. usually client has connection pooling already. unless conn pooling is not utilized, than 100k is still probably not enough – Tuan Anh Tran Oct 12 '20 at 09:38

1 Answers1

4

The error message max number of clients reached is clearly indicate that Redis is reached client limit and unable to serve any new requests.

  • this issue probably can be related to incorrect use of Predis\Client in code. Instead to create a connection object once (singleton) and use it across process lifetime. The code probably create a new object on every request to Redis and keep all these connections open.
  • another thing is worth to check how php processes are managed by a web server. The web server (e.g. apache prefork, nginx php-fpm) might leave processes for a long time both holding connections to Redis and exhaust server resources (mem, cpu).
  • if nothing from above is true - the issue (bug) might be in the predis library.

Bottom line: the code/web server exhaust maxclients limit.

If you don't have control over code/web server (e.g. nginx), to reduce amount of error messages you can:

  • increase maxclients over 10k (depends on your Redis server resources). This will reduce frequency of error messages.
  • consider to enable (disabled by default) connection timeout (use it with cautious, as your code may assume that connections are never timeout). This will release old connections from a connection pool.
  • decrease tcp-keepalive from 300 seconds to less than timeout. This will close connections to dead peers (clients that cannot be reached even if they look connected).
Dharman
  • 30,962
  • 25
  • 85
  • 135
Tom Lime
  • 1,154
  • 11
  • 15
  • thank you Tom! this is super valueable info but it doesn't answer my question. I want to change the returned error message... – silver Oct 16 '20 at 07:50
  • 1
    Haha! Thats true! I am still not able to understand - are you get this error message in log (app/redis) or as output from a php process? – Tom Lime Oct 16 '20 at 07:55
  • my php app get this message when trying to connect to the redis server. – silver Oct 16 '20 at 08:00
  • Got it. In this case if you don't care about all errors. You can disable them - https://www.php.net/manual/en/errorfunc.configuration.php#ini.error-reporting Or if you have access to the code you can check if `predis` supports custom logging per object instance. – Tom Lime Oct 16 '20 at 08:07
  • not good, my app get this error and use the returned string as a valid result which breaks my logic... – silver Oct 16 '20 at 08:11
  • @silver could you not create a Predis wrapper in your PHP application (create your own `Cache` class which wraps calls like `get`, `put`, `delete`) and then wrap each of these in a try catch? If the exception back matches the max connections message, do nothing, otherwise throw the exception (you could also check the exception code to make it slightly more robust, checking strings can sometimes be brittle) – Halfpint Oct 19 '20 at 08:12
  • 1
    Either way I think @TomLime answer is very pescriptive on how to actually remedy the error. It's not normal behaviour in an application to receive this message (I'm not sure how large your application is, but we have multiple webservers behind a load balancer and we make hundreds and sometimes thousands of calls to redis per minute through a singleton on each server, we never get this isseu) It sounds to me like you need to invest some time in properly handling connections to your Redis cluster opposed to silently catching the error (silent catching is pretty dangerous too imo) – Halfpint Oct 19 '20 at 08:14