37

I'm using Redis in my application, both for Sidekiq queues, and for model caching.

What is the best way to have a Redis connection available to my models, considering that the models that will be hitting Redis will be called both from my Web application (ran via Puma), and from background jobs inside Sidekiq?

I'm currently doing this in my initializers:

Redis.current = Redis.new(host: 'localhost', port: 6379)

And then simply use Redis.current.get / Redis.current.set (and similar) throughout the code...

This should be thread-safe, as far as I understand, since the Redis Client only runs one command at a time, using a Monitor.

Now, Sidekiq has its own connection pool to Redis, and recommends doing

Sidekiq.redis do |conn|
   conn.get
   conn.set
end

As I understand it, this would be better than the approach of just using Redis.current because you don't have multiple workers on multiple threads waiting on each other on a single connection when they hit Redis.

However, how can I make this connection that I get from Sidekiq.redis available to my models? (without having to pass it around as a parameter in every method call)

I can't set Redis.current inside that block, since it's global, and I'm back to everyone using the same connection (plus switching between them randomly, which might even be non-thread-safe)

Should I store the connection that I get from Sidekiq.Redis into a Thread-local variable, and use that thread-local variable everywhere?

In that case, what do I do in the "Puma" context? How do I set the thread-local variable?

Any thoughts on this are greatly appreciated.

Thank you!

Daniel Magliola
  • 30,898
  • 61
  • 164
  • 243

1 Answers1

52

You use a separate global connection pool for your application code. Put something like this in your redis.rb initializer:

require 'connection_pool'
REDIS = ConnectionPool.new(size: 10) { Redis.new }

Now in your application code anywhere, you can do this:

REDIS.with do |conn|
  # some redis operations
end

You'll have up to 10 connections to share amongst your puma/sidekiq workers. This will lead to better performance since, as you correctly note, you won't have all the threads fighting over a single Redis connection.

All of this is documented here: https://github.com/mperham/sidekiq/wiki/Advanced-Options#connection-pooling

Mike Perham
  • 21,300
  • 6
  • 59
  • 61
  • 3
    Now, in this case, I'd have my own connection pool for my code (in your example, 10 connections), Sidekiq will have its own connection pool, which it's going to use for its "workers work", and i'm not going to be using "Sidekiq.redis" anywhere in my code. Is this correct? – Daniel Magliola Feb 03 '15 at 10:22
  • This worked beautifully in combination with Puma, Redis and Sidekiq. My front-end subscribes to an event-stream in one of my controllers and is much, much more performant now. Like @DanielMagliola said, I never use Sidekiq.redis in my code, you refer to the REDIS variable that's available app wide. – Webdevotion Apr 14 '15 at 10:40
  • @MikePerham so then do we initialize sidekiq using `REDIS` too? ` Sidekiq.configure_server do |config| config.redis = REDIS end Sidekiq.configure_client do |config| config.redis = REDIS end ` – eggie5 Sep 09 '15 at 17:14
  • The Sidekiq wiki describes how to initialize Redis. You can give it a connection pool directly but that's an advanced option you should only use if you know what you are doing. – Mike Perham Sep 09 '15 at 18:45
  • @MikePerham if we must have the Redis pool already available for other initializers, would you recommend defining the REDIS constant in `config.before_initialize` in application.rb ? – sandre89 Oct 12 '19 at 12:18