1

So I need to be able to create a single RabbitMQ connection and channel per Sidekiq thread because I run out of RabbitMQ connections if I don't and because the docs suggest it. The docs show how to do it with Unicorn:

before_fork do |server, worker|
  $rabbitmq_connection.close if $rabbitmq_connection
end

after_fork do |server, worker|
  # the following is *required* for Rails + "preload_app true",
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection

    $rabbitmq_connection = Bunny.new
    $rabbitmq_connection.start

    $rabbitmq_channel    = $rabbitmq_connection.create_channel
  end
end

Is it possible to do something similar for Sidekiq threads? Is there something I can do in Sidekiq.server_configure? It looks like this is where Sidekiq starts a thread but I don't see anyway to hook into the start/stop?

gabe
  • 1,873
  • 2
  • 20
  • 36

1 Answers1

2

Create your own pool of connections to RabbitMQ, as detailed here:

https://github.com/mperham/sidekiq/wiki/Advanced-Options#connection-pooling

Mike Perham
  • 21,300
  • 6
  • 59
  • 61
  • If I'm using the default Sidekiq concurrency of 25, should I set the `ConnectionPool` `size` to 25 i.e. `ConnectionPool.new(size: Sidekiq.options[:concurrency], timeout: 3) { LaGear::Bus.new }`? – gabe Jan 11 '15 at 23:35
  • Generally yes unless you want to limit the connections for some reason. – Mike Perham Jan 11 '15 at 23:49
  • So if I'm using 25 for the concurrency does Sidekiq spin-up 25 threads per worker? – gabe Jan 14 '15 at 00:33
  • I don't know what `worker` means. 25 threads per process, yes. – Mike Perham Jan 14 '15 at 04:39
  • `worker` as in my app has 2 Sidekiq workers `MyFasterJobWorker` and `MyPainfullyLongJobWorker`. Ok, so it's 25 threads total for the Sidekiq process and not per `worker`. Ok, so if I have multiple workers defining a `ConnectionPool` constant for the same resource, it seems like I should set the `ConnectionPool` `size` to less per worker and not to `Sidekiq.options[:concurrency]`. If I have 2 `workers`, wouldn't that be 2 * 25 connections = 50 connections for 2 workers (I'm wanting 1 per thread)? I'm probably misunderstanding something fundamental about `connection_pool`. – gabe Jan 14 '15 at 04:55
  • 1
    Define the pool in your initializer. Both workers can use it. 25 threads means you'll need a max of 25 connections if you are processing 25 jobs at once. – Mike Perham Jan 14 '15 at 05:33