Why am I getting "OpenSSL::SSL::SSLError: SSL_read: sslv3 alert bad record mac" intermittently from Heroku Redis?

Question

Since switching from Redis To Go to Heroku Redis, the Redis code in our Ruby on Rails app gets an "OpenSSL::SSL::SSLError: SSL_read: sslv3 alert bad record mac" error a few times a day.

Any ideas why?

Answer 1

I believe you are running into a multiprocessing issue where a forked process closes the Redis connection of the parent process. I have just found a bug that lead to the same error in resque which suffered from this problem.

https://github.com/resque/resque/pull/1739

Answer 2

It doesn't solve the issue directly, but this is what Heroku support had to say:

This issue has been a difficult one to diagnose due to it's infrequency and inconsistency. We have many reports of it occurring that span destinations, app languages, Heroku Dyno configurations, and plenty of other details. This issue has been written up and is being diagnosed by engineers, but again the details have made it almost impossible for us to do so.

Additionally, we do not have infrastructure managing outgoing connections. We do have network usage information in raw forms (bytes transferred, number of packets, etc.), but unlike incoming connections which has the Heroku Router dealing with requests, outbound connections are all "standard" as provided by our infrastructure provider. There is no Heroku-specific infrastructure that deals with outbound connections. The only item of interest there is the virtual interface Dynos use, and by extension the Dyno host's network configuration, but again there is nothing special about this. It uses the infrastructure platform provided network configuration necessary for host communication.

Neither myself nor engineers have come up with a concise answer for these issues so far, given their inconsistency our current recommendation is that these issues are better handled with connection error handling, logging as needed, and retrying.

If you have details on a consistently reproducible way this error occurs it would aid us significantly.

Answer 3

I've worked around this in a Ruby on Rails app by retrying these errors. So far it seems to have gotten rid of the errors. We used to get about 5 a day, and in the past day – since introducing this workaround – there have been none.

In config/initializers/redis_heroku_error_handling.rb:

# Hack to retry errors on Heroku Redis: https://stackoverflow.com/questions/50228454/why-am-i-getting-opensslsslsslerror-ssl-read-sslv3-alert-bad-record-mac

raise "Ensure this hack still works!" if Redis::VERSION != "3.3.5"

module RedisHerokuErrorHandlingHack
  def call(command)
    attempts_left = 3

    begin
      super
    rescue OpenSSL::SSL::SSLError => e
      raise unless e.message.include?("SSL_read: sslv3 alert bad record mac")

      attempts_left -= 1
      raise unless attempts_left > 0

      retry
    end
  end
end

class Redis::Client
  prepend RedisHerokuErrorHandlingHack
end

In spec/initializers/redis_heroku_error_handling_spec.rb:

require "rails_helper"

describe RedisHerokuErrorHandlingHack do
  it "retries 'bad record mac' errors" do
    exception = OpenSSL::SSL::SSLError.new("SSL_read: sslv3 alert bad record mac")
    fail_then_maybe_succeed(exception: exception, fail_times: 2)

    expect($redis.get("hello")).to eq("success response")
  end

  it "fails if a few retries didn't help" do
    exception = OpenSSL::SSL::SSLError.new("SSL_read: sslv3 alert bad record mac")
    fail_then_maybe_succeed(exception: exception, fail_times: 3)

    expect { $redis.get("hello") }.to raise_error(/bad record mac/)
  end

  it "does not retry other errors" do
    fail_then_maybe_succeed(exception: "Boom", fail_times: 2)

    expect { $redis.get("hello") }.to raise_error("Boom")
  end

  private

  def fail_then_maybe_succeed(exception:, fail_times:)
    attempt_count = 0

    allow_any_instance_of(Redis::Client).to receive(:process) do |*args|
      attempt_count += 1

      if attempt_count <= fail_times
        raise exception
      else
        "success response"
      end
    end
  end
end