How to protect against Redis::TimeoutError: Connection timed out on Heroku

Question

This might seem like a silly question, but I figured someone here on StackOverflow might have some ideas around this. What the hell, right?

I'm using Heroku workers with 1X Dynos to run Resque. Sometimes I get this error: Redis::TimeoutError: Connection timed out. It happens in the redis gem; here's the stacktrace:

Redis::TimeoutError Connection timed out
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/connection/ruby.rb:55 rescue in _read_from_socket
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/connection/ruby.rb:48 _read_from_socket
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/connection/ruby.rb:41 gets
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/connection/ruby.rb:273 read
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:245 block in read
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:233 io
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:244 read
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:175 block in call_pipelined
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:214 block (2 levels) in process
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:340 ensure_connected
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:204 block in process
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:286 logging
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:203 process
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:174 call_pipelined
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:146 block in call_pipeline
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:273 with_reconnect
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis/client.rb:144 call_pipeline
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis.rb:2101 block in pipelined
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis.rb:37 block in synchronize
    vendor/ruby-2.1.5/lib/ruby/2.1.0/monitor.rb:211 mon_synchronize
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis.rb:37 synchronize
    vendor/bundle/ruby/2.1.0/gems/redis-3.2.0/lib/redis.rb:2097 pipelined
    vendor/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:413 namespaced_block
    vendor/bundle/ruby/2.1.0/gems/redis-namespace-1.5.1/lib/redis/namespace.rb:265 pipelined
    vendor/bundle/ruby/2.1.0/gems/resque-1.25.2/lib/resque.rb:214 push
    vendor/bundle/ruby/2.1.0/gems/resque-1.25.2/lib/resque/job.rb:153 create
    vendor/bundle/ruby/2.1.0/gems/resque_solo-0.1.0/lib/resque_ext/job.rb:7 create_solo
    vendor/bundle/ruby/2.1.0/gems/resque-1.25.2/lib/resque.rb:317 enqueue_to
    vendor/bundle/ruby/2.1.0/gems/resque-1.25.2/lib/resque.rb:298 enqueue

We have resque-retry set up, but I guess it doesn't matter if the enqueue call can't even connect to redis.

My current solution is to wrap every Resque.enqueue call with begin/rescue so that we can re-try the enqueue call (as per https://github.com/resque/resque/issues/840). But isn't there a better way?

score 4 · Accepted Answer · answered Nov 16 '15 at 07:38

Heroku Redis allows you to change your instance timeout and maxmemory-policy settings. Theses settings will be kept across upgrades and HA failover.

From documentation:

The timeout setting sets the number of seconds Redis waits before killing idle connections. A value of zero means that connections will not be closed. The default value is 300 seconds (5 minutes). You can change this value using the CLI:
$ heroku redis:timeout maturing-deeply-2628 --seconds 60
Timeout for maturing-deeply-2628 (REDIS_URL) set to 60 seconds. Connections to the redis instance will be stopped after idling for 60 seconds.

Looks like in case of resque the 0 (do not use connection time out) would be the best choise (--seconds 0).

How to protect against Redis::TimeoutError: Connection timed out on Heroku

1 Answers1