0

Update: From the below note it appears that Azure Redis has a 60 second idle time for its configuration. The connectionwatchdog in redis is seeing the connection and reporting correctly the disconnect about every 2 minutes which makes sense i believe with the 60 second settings. My question now is how to keep the connection for PubSub in Lettuce open? It was suggested to do a periodic PING. Lettuce forbids this in the following class: PubSubEndpoint line 144 method validateCommandAllowed()

Link to Project that can produce problem Link to logs

Thank you in advance, David

Community
  • 1
  • 1
David Parry
  • 1
  • 2
  • Hi David, welcome to Stack Overflow. This looks like an issue between `maxclients:1000` and `lettuce.connection.pool.total.max=300`. Try not using the pool but a single connection instead, – mp911de Jun 11 '19 at 19:02
  • That did not help. I am pretty certain that ConnectionWatchdog is created to listen to my RedisPubSubListener impl that is added to the PubSub channel that i have subscribed to in my application. From my reading I believe the watchdog purpose is to keep the connection to the channel. So it could be just information and perfectly fine. That is my question first is this message a concern or normal. If it is not normal than some direction to eliminate this message: network, firewall. My local install on docker no error and prior job Redis Enterprise did not exhibit this either only Azure does? – David Parry Jun 12 '19 at 13:41
  • ConnectionWatchdog listens for disconnects and re-connects a disconnected connection. A disconnection has always a reason and that’s where my suggestion was heading. Experiencing a larger number of disconnects is investigationworthy (anything greater 1 disconnection per hour) – mp911de Jun 12 '19 at 14:21
  • @mp911de thank you for your input. My assumption this is not the lettuce api correct? Instead something in our network causing the connection to be lost and the ConnectionWatchdog is doing its job! I was wondering if you had some pointers to help the network guys at Azure to help resolve this? – David Parry Jun 14 '19 at 18:02
  • Just to keep the update going. We have a ticket open with Azure. Also installed our own Redis in Azure with same docker springboot image just pointing to our installed Redis and the ConnectionWatchdog has not reported a problem for over 2 days. My conclusion is the network between our springboot microservice and Azure Redis in the cloud is not stable. @mp911de or anyone else have an opinion? – David Parry Jun 17 '19 at 16:37
  • Asked to assist and give example of the reconnect and watchdog in action. Here is a link to the complete SpringBoot app in github https://github.com/davidparry/azure-redis-watchdog any feedback would be awesome thank you all for your help. – David Parry Jun 19 '19 at 02:17
  • Hi David, is it possible that you are hitting client idle timeout? did you try running your local redis-server with --timeout config option to see it repros? – Deepak Jun 20 '19 at 23:44
  • When I try running the repro app I can see there's no activity on the connection (by running monitor command) which causes redis to close an idle connection (for Azure it's 60 seconds). Does lettuce have a setting to send periodic pings? If not then application would have to send periodic pings to prevent the connection from getting into idle state. client list command can be used to see how long a connection has been idle. hth. – Deepak Jun 21 '19 at 00:37
  • @mp911de any help would appreciate it. I tried to ping like suggested but get Command PING not allowed while subscribed to a channel. In PubSubEndpoint line 144 method validateCommandAllowed will not allow the PING to be sent. I have tried timeout ZERO on connection too. But i can not seem to figure out how to keep the connection alive when the server is set to 60 seconds. – David Parry Jun 21 '19 at 03:30
  • one correction, for azure it's 600 seconds. Also I see from the client list there are two connections, one is pub-sub and second one is normal. redis-server doesn;t close pubsub connections, but the other one is closed when it reaches the idle timeout. – Deepak Jun 21 '19 at 18:38
  • @Deepak the StatefulRedisPubSubConnection in Springboot which was not being used to subscribe to a Channel does not test the connection to keep it open. Instead it waits for watchdog or i would have to do a PING in my code which is not desired. The answer is remove the unused PubSub until you need it. – David Parry Jun 28 '19 at 21:11

0 Answers0