0

Under heavy load on the application server, occasionally we are getting following error while using CacheManager.NET and Redis as backend,

Redis Connection Timeout Exception :Timeout performing SCRIPT, inst: 0, mgr: CheckForStaleConnections, err: never, queue: 11596, qu: 11596, qs: 0, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, clientName: HOST, serverEndpoint: Unspecified/SREDIS01:6379, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=0,Free=2047,Min=4,Max=2047), Local-CPU: 4.84% (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md), While GetOrAdd key :XYZ|

Although we have setup the SyncTimeout to 10000 (10 seconds) and ConnectTimeout to 5 seconds in ConnectionMultiplexer, the system gets stuck for around 50 to 60 minutes (even more sometimes). Please note that this issue comes only when executing SCRIPT commands, which Cache Manager executes in the initialization.

Stack Trace is :

   at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
   at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
   at StackExchange.Redis.RedisServer.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
   at StackExchange.Redis.RedisServer.ScriptLoad(String script, CommandFlags flags)
   at StackExchange.Redis.LuaScript.Load(IServer server, CommandFlags flags)
   at CacheManager.Redis.RedisCacheHandle`1.LoadScripts()
   at CacheManager.Redis.RedisCacheHandle`1.Eval(ScriptType scriptType, RedisKey redisKey, RedisValue[] values, CommandFlags flags)
   at CacheManager.Redis.RedisCacheHandle`1.<>c__DisplayClass48_0.<GetCacheItemAndVersion>b__0()
   at CacheManager.Redis.RetryHelper.Retry[T](Func`1 retryme, Int32 timeOut, Int32 retries, ILogger logger)
   at CacheManager.Redis.RedisCacheHandle`1.Retry[T](Func`1 retryme)
   at CacheManager.Redis.RedisCacheHandle`1.GetCacheItemAndVersion(String key, String region, Int32& version)
   at CacheManager.Redis.RedisCacheHandle`1.GetCacheItemInternal(String key, String region)
   at CacheManager.Core.Internal.BaseCache`1.GetCacheItem(String key, String region)
   at CacheManager.Core.BaseCacheManager`1.GetCacheItemInternal(String key, String region)
   at CacheManager.Core.Internal.BaseCache`1.GetCacheItem(String key, String region)
   at CacheManager.Core.Internal.BaseCache`1.Get(String key, String region)

I went through Timeout documentation of StackExchange Redis and can understand that due to very heavy CPU usage on the client machine, some of the connections may run in to this types of issue.

However our big concern is it should get timed out in 10 seconds and should not stuck for minutes / hours.

Can you please guide us here.

jatanp
  • 3,982
  • 4
  • 40
  • 46
  • In general, try lower the number of retries and retry timeout, maybe even disable retries at all and see what kind of exception you get. Maybe there is something wrong with your setup and the build in retry logic actually cannot solve/work with that... or just handle retries yourself (try/catch). Also, if the process does retry for too long/too many tries, you might see those issues and just end producing even more load on the server... – MichaC Oct 23 '17 at 07:57
  • @MichaC you are right, I had to go through your and stackexchange redis code, and finally found the issue in our configuration code which configures ConnectionMultiplexer and CacheManager. We had abnormally high retries (i.e. 1000), that could be due to typo. I was able to calculate the effective delay because of combination of synctimeout and timeout + retry of cachemanager configuration. Thanks for the pointers, and very nice coding in CacheManager – jatanp Oct 23 '17 at 20:03

0 Answers0