Heroku, Rails, Resque, Redis: All jobs failing because of Errno::ETIMEDOUT

Question

I have a rails app on heroku that uses Resque.enqueu to schedule background jobs using Redis and redistogo.

Jobs used to work normally but I notice now that they have all been failing for some time (a few weeks, 100% of jobs).

The error I get:

Exception Errno::ETIMEDOUT Error Connection timed out - connect(2)

And the stack trace:

> /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:878:in `initialize'
> /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:878:in `open'
> /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:878:in `block in
> connect' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/timeout.rb:52:in
> `timeout' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:877:in
> `connect' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:862:in
> `do_start' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:851:in
> `start' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1373:in
> `request' /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1131:in
> `get'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/connection.rb:121:in `block in request'
> /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.1/lib/active_support/notifications.rb:164:in
> `block in instrument'
> /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.1/lib/active_support/notifications/instrumenter.rb:20:in
> `instrument'
> /app/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.1/lib/active_support/notifications.rb:164:in
> `instrument'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/connection.rb:118:in `request'
> /app/vendor/bundle/ruby/2.0.0/gems/shopify_api-4.0.6/lib/active_resource/connection_ext.rb:13:in `request_with_detailed_log_subscriber'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/connection.rb:82:in
> `block in get'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/connection.rb:216:in `with_auth'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/connection.rb:82:in
> `get'
> /app/vendor/bundle/ruby/2.0.0/gems/activeresource-4.0.0/lib/active_resource/custom_methods.rb:57:in
> `get'
> /app/vendor/bundle/ruby/2.0.0/gems/shopify_api-4.0.6/lib/shopify_api/countable.rb:4:in
> `count' /app/app/models/shop.rb:263:in `unscanned_customers'
> /app/app/models/shop.rb:230:in `shopify_customers'
> /app/app/models/shop.rb:144:in `bulk_scan'
> /app/app/workers/bulk_scanner.rb:16:in `perform'

I was thinking maybe it is something to do with not connecting to Redis or redistogo properly. Here are the relevant initializer code that I have:

resque.rb:

Resque.redis = RedisConnection.connection

redis_connection.rb:

class RedisConnection
  def self.close
    connection.quit
  end

  def self.connection
    @connection ||= new_connection
  end

  def self.new_connection
    uri = URI.parse(ENV['REDISTOGO_URL'] || 'redis://localhost:6379/')

    Redis.new(host: uri.host,
              port: uri.port,
              password: uri.password,
              thread_safe: true,
              :timeout => 1)
  end
end

If I call Resque.redis or @connection in my app, it returns a Redis instance. What else would be causing this connection error? Where can I troubleshoot?

natemacinnes · Answer 1 · 2016-12-01T21:34:13.863

Looking at the Backtrace, could it be a Shopify API connection timeout? It looks like unscanned_customers is a supposed to be a countable Shopify collection and the problem occurs when an attempt to retrieve the customer is made.

I would start my debugging here: > 'count' /app/app/models/shop.rb:263:in 'unscanned_customers'

Additionally, I would attempt to connect to Shopify from the server with a private app API key and password:

shop_url = "https://#{API_KEY}:#{PASSWORD}@SHOP_NAME.myshopify.com/admin"
ShopifyAPI::Base.site = shop_url

Refer to the Shopify API for more details if needed Shopify API Readme

It will help isolate the connection issue.

Cheers.

Heroku, Rails, Resque, Redis: All jobs failing because of Errno::ETIMEDOUT

1 Answers1