sidekiq - runaway FIFO pipes created with large job

Question

We are using Sidekiq to process a number of backend jobs. One in particular is used very heavily. All I can really say about it is that it sends emails. It doesn't do the email creation (that's a separate job), it just sends them. We spin up a new worker for each email that needs to be sent.

We are trying to upgrade to Ruby 3 and having problems, though. Ruby 2.6.8 has no issues; in 3 (as well as 2.7.3 IIRC), if there is a large number of queued workers, it will get through maybe 20K of them, then it will start hemorrhaging FIFO pipes, on the order of 300-1000 ever 5 seconds or so. Eventually it gets to the ulimit on the system (currently set at 64K) and all sockets/connections fail due to insufficient resources.

In trying to debug this issue I did a run with 90% of what the email worker does entirely commented out, so it does basically nothing except make a couple database queries and do some string templating. I thought I was getting somewhere with that approach, as one run (of 50K+ emails) succeeded without the pipe explosion. However, the next run (identical parameters) did wind up with the runaway pipes.

Profiling with rbspy and ruby-prof did not help much, as they primarily focus on the Sidekiq infrastructure, not the workers themselves.

Looking through our code, I did see that nothing we wrote is ever using IO.* (e.g. IO.popen, IO.select, etc), so I don't see what could be causing the FIFO pipes.

I did see https://github.com/mperham/sidekiq/wiki/Batches#huge-batches, which is not necessarily what we're doing. If you look at the code snippet below, we're basically creating one large batch. I'm not sure whether pushing jobs in bulk as per the link will help with the problem we're having, but I'm about to give it a try once I rework things a bit.

No matter what I do I can't seem to figure out the following:

What is making these pipes? Why are they being created? What is the condition by which the pipes start getting made exponentially? There are two FIFO pipes that open when we start Sidekiq, but until enough work has been done, we don't see more than 2-6 pipes open generally. Any advice is appreciated, even along the lines of where to look next, as I'm a bit stumped.

Initializer:

require_relative 'logger'
require_relative 'configuration'
require 'sidekiq-pro'
require "sidekiq-ent"

module Proprietary
  unless const_defined?(:ENVIRONMENT)
    ENVIRONMENT = ENV['RACK_ENV'] || ENV['RAILS_ENV'] || 'development'
  end
  # Sidekiq.client_middleware.add Sidekiq::Middleware::Client::Batch
  REDIS_URL = if ENV["REDIS_URL"].present?
                ENV["REDIS_URL"]
              else
                "redis://#{ENV["REDIS_SERVER"]}:#{ENV["REDIS_PORT"]}"
              end

  METRICS = Statsd.new "10.0.9.215", 8125

  Sidekiq::Enterprise.unique! unless Proprietary::ENVIRONMENT == "test"
  Sidekiq.configure_server do |config|
    # require 'sidekiq/pro/reliable_fetch'
    config.average_scheduled_poll_interval = 2
    config.redis = {
      namespace: Proprietary.config.SIDEKIQ_NAMESPACE,
      url: Proprietary::REDIS_URL
    }
    config.server_middleware do |chain|
      require 'sidekiq/middleware/server/statsd'
      chain.add Sidekiq::Middleware::Server::Statsd, :client => METRICS
    end
    config.error_handlers << Proc.new do |ex,ctx_hash|
      Proprietary.report_exception(ex, "Sidekiq", ctx_hash)
    end
    config.super_fetch!
    config.reliable_scheduler!
  end

  Sidekiq.configure_client do |config|
    config.redis = {
      namespace: Proprietary.config.SIDEKIQ_NAMESPACE,
      url: Proprietary::REDIS_URL,
      size: 15,
      network_timeout: 5
    }
  end
end

Code snippet (sanitized)

  def add_targets_to_batch
    @target_count  = targets.count
    queue_counter = 0
    batch.jobs do
      targets.shuffle.each do |target|
        send(campaign_target)
        queue_counter += 1
      end
    end
  end

  def send(campaign_target)
    TargetEmailWorker.perform_async(target[:id],
                                            guid,
                                            is_draft ? target[:email_address] : nil)
      begin
        Target.where(id: target[:id]).update(send_at: Time.now.utc)
      rescue Exception => ex
        
        Proprietary.report_exception(ex, self.class.name, { target_id: target[:id], guid: guid })
      end
    end
  end

First I tried auditing our external connections for connection pooling, etc. That did not help the issue. Eventually I got to the point where I disabled all external connections and let the job run doing virtually nothing outside of a database query and some logging. This allowed one run to complete without issue, but on the second one, the FIFO pipes still grew exponentially after a certain (variable) amount of work was done.

sidekiq - runaway FIFO pipes created with large job

0 Answers0