Singleton disappears when forking unicorn processes in Rails

Question

I've got a rails app running ruby 2.4.4 using Unicorn as a web server which makes use of a singleton to read from Kafka in a background thread. The idea is to have a single instance of the singleton per unicorn process. So 4 processes, 4 singletons.

I kick off the kafka consumption inside the after_fork hook in my unicorn config. I can successfully wait for the consumption of historic messages to complete (verified by putting a pry immediately after).

However when I get to the point of serving traffic, the singleton instance is a) a different instance, and b) empty - the ivar previously set is gone.

I have confirmed that I am inside the same process and the same thread.

The setup is as follows:

# background_foo_consumer.rb
class BackgroundFooConsumer
  include Singleton

  attr_reader :background_consumer

  def add_background_consumer(consumer, topics, options: nil)
    @background_consumer ||= BackgroundKafkaConsumer.new(consumer, topics, options: options)
  end

  def processed_historical_messages?
    background_consumer&.consumer&.reached_head
  end
end


# config/unicorn.rb
after_worker_ready do |server, worker|
  BackgroundFooConsumer.instance.add_background_consumer(nil, ["foos"])
  BackgroundFooConsumer.instance.background_consumer.start

  BackgroundFooConsumer.instance.background_consumer.consumer.mutex.synchronize {
    BackgroundFooConsumer
    .instance.background_consumer.consumer.processed_historical_messages.wait(
      BackgroundFooConsumer.instance.background_consumer.consumer.mutex
    )
  }
  end
end

I confirmed I am inside the same process, even the same thread, as I can successfully pass the correct object through to the application by replacing include Singleton with a custom implementation and Thread local variables as follows:

# config/unicorn.rb
after_worker_ready do |server, worker|
  # ... same as above

  Thread.current[:background_foo_consumer] = BackgroundFooConsumer.instance
end


# background_foo_consumer.rb
class BackgroundFooConsumer
  attr_reader :background_consumer

  def self.instance
    @instance ||= begin
                    Thread.current[:background_foo_consumer] || self.new
                  ensure
                    Thread.current[:background_foo_consumer] = nil
                  end
  end
end

In this implementation, when I come to serve traffic from my app BackgroundFooConsumer.instance is the correct instance created in the after_fork hook, and there is an independent instance per unicorn process, confirmed by checking the object id.

I don't believe this is the GC, at least the underlying object does not get mopped up, I have confirmed this by setting the Thread local variable in the after_fork hook, but then using include Singleton in my consumer class. I still get the empty/new singleton, but the thread local variable is still present if I query it directly.

My current hypothesis is that this is something to do with copy on write, and by setting the thread local variables I somehow force ruby to create me a singleton for that process only and save it to that variable.

So my question is how can a singleton instance disappear like this inside a single thread? And how can I stop it from happening? I'd prefer not to use these thread local variables if I can help it.

Forking does not mean sharing memory afaik, everything is copied. — Joel Blum, Jun 02 '20 at 08:26
For sure, as soon as you write anything to the shared memory it's copied, but I don't want to share memory. I want the memory to be allocated for the singleton for each process independently. — timpwbaker, Jun 02 '20 at 08:31

score 0 · Accepted Answer · answered Jun 03 '20 at 11:31

The answer to this ended up being due to a pretty niche bit of rails configuration: cache_classes. I was running my unicorn server locally and so the classes were not being cached.

Rails (when running in anything other than production mode, which is generally used in both staging and production but not local) reloads class level objects if they change that are otherwise static in production.

Effectively rails was seeing some change and reloading the classes as this prevents the programmer from having the restart the server.

This is controlled by a bit of config called cache_classes - I had heard of it before, it’s the reason why you need to restart the server after running a migration in production in order for any changes to be accessible from the ActiveRecord objects. I didn’t put two and two together on this manifestation though as I didn’t know the classes would be reloaded. I'm still unsure why they'd be considered changed and in need of reloading.

Ultimately I wouldn’t have seen this problem if I wasn’t trying to run the unicorn server locally and it can be prevented by setting config.cache_classes = true in development.rb

Docs here: https://guides.rubyonrails.org/configuring.html#rails-general-configuration

Singleton disappears when forking unicorn processes in Rails

1 Answers1