Since switching from Redis To Go to Heroku Redis, the Redis code in our Ruby on Rails app gets an "OpenSSL::SSL::SSLError: SSL_read: sslv3 alert bad record mac" error a few times a day.
Any ideas why?
Since switching from Redis To Go to Heroku Redis, the Redis code in our Ruby on Rails app gets an "OpenSSL::SSL::SSLError: SSL_read: sslv3 alert bad record mac" error a few times a day.
Any ideas why?
I believe you are running into a multiprocessing issue where a forked process closes the Redis connection of the parent process. I have just found a bug that lead to the same error in resque
which suffered from this problem.
It doesn't solve the issue directly, but this is what Heroku support had to say:
This issue has been a difficult one to diagnose due to it's infrequency and inconsistency. We have many reports of it occurring that span destinations, app languages, Heroku Dyno configurations, and plenty of other details. This issue has been written up and is being diagnosed by engineers, but again the details have made it almost impossible for us to do so.
Additionally, we do not have infrastructure managing outgoing connections. We do have network usage information in raw forms (bytes transferred, number of packets, etc.), but unlike incoming connections which has the Heroku Router dealing with requests, outbound connections are all "standard" as provided by our infrastructure provider. There is no Heroku-specific infrastructure that deals with outbound connections. The only item of interest there is the virtual interface Dynos use, and by extension the Dyno host's network configuration, but again there is nothing special about this. It uses the infrastructure platform provided network configuration necessary for host communication.
Neither myself nor engineers have come up with a concise answer for these issues so far, given their inconsistency our current recommendation is that these issues are better handled with connection error handling, logging as needed, and retrying.
If you have details on a consistently reproducible way this error occurs it would aid us significantly.
I've worked around this in a Ruby on Rails app by retrying these errors. So far it seems to have gotten rid of the errors. We used to get about 5 a day, and in the past day – since introducing this workaround – there have been none.
In config/initializers/redis_heroku_error_handling.rb
:
# Hack to retry errors on Heroku Redis: https://stackoverflow.com/questions/50228454/why-am-i-getting-opensslsslsslerror-ssl-read-sslv3-alert-bad-record-mac
raise "Ensure this hack still works!" if Redis::VERSION != "3.3.5"
module RedisHerokuErrorHandlingHack
def call(command)
attempts_left = 3
begin
super
rescue OpenSSL::SSL::SSLError => e
raise unless e.message.include?("SSL_read: sslv3 alert bad record mac")
attempts_left -= 1
raise unless attempts_left > 0
retry
end
end
end
class Redis::Client
prepend RedisHerokuErrorHandlingHack
end
In spec/initializers/redis_heroku_error_handling_spec.rb
:
require "rails_helper"
describe RedisHerokuErrorHandlingHack do
it "retries 'bad record mac' errors" do
exception = OpenSSL::SSL::SSLError.new("SSL_read: sslv3 alert bad record mac")
fail_then_maybe_succeed(exception: exception, fail_times: 2)
expect($redis.get("hello")).to eq("success response")
end
it "fails if a few retries didn't help" do
exception = OpenSSL::SSL::SSLError.new("SSL_read: sslv3 alert bad record mac")
fail_then_maybe_succeed(exception: exception, fail_times: 3)
expect { $redis.get("hello") }.to raise_error(/bad record mac/)
end
it "does not retry other errors" do
fail_then_maybe_succeed(exception: "Boom", fail_times: 2)
expect { $redis.get("hello") }.to raise_error("Boom")
end
private
def fail_then_maybe_succeed(exception:, fail_times:)
attempt_count = 0
allow_any_instance_of(Redis::Client).to receive(:process) do |*args|
attempt_count += 1
if attempt_count <= fail_times
raise exception
else
"success response"
end
end
end
end