Why isn't unicorn+mysql2 gem showing latest version of model data on multi-worker prod setup?

Question

My model doesn't seem to be updated correctly.

• Unicorn rainbows! on EventMachine
• mysql2 gem

I can reproduce in the production environment with 1000 rpm on 8 workers. If I update my_model in the Rails console (with a call to reload!), all works fine. Locally I can't reproduce it.

From the controller:

# params[:my_model] = {:name => "new name"} 
def update
  @my_model = MyModel.first # {:name => "old name"}
  Rails.logger.info @my_model.name
  @my_model.update_attributes(params[:my_model])
  redirect_to :action => :index
end

Log:

new name
old name
old name
old name
new name

What do I wrong? Thanks for advance!

Answer 1

Is the data in the model being cached by some of the workers, so that the data appears to be the new name in one worker, but the old name in another worker?

It could be helpful to add per-work IDs, and output these to your log. In other words, when your worker starts up it should assign itself an ID that is unique from the other workers (the worker's PID would be a good candidate). This way you could see if one or more workers consistently have the problem, if it's all workers, or if it's only some workers some of the time. If this turns out to be a caching problem it could be that you just need to wait for the catch to be invalidated on all workers for that object, and that it will eventually show the correct, updated value for all workers.

If you can't reproduce it locally, is it because you're only using a single web server instance (one worker) when developing locally?

Also, check you production logs and see if the word CACHE is showing up in the instances where the old name is being shown.

Answer 2

It looks like root cause is that sql cache doesn't cleared between actions when server is under high load as worker processes doesn't go out action.
I tested on server without load - problem isn't reproduced. Code below from ApplicationController fixed this problem, but this solution a bit ugly:

  before_filter do 
    MyModel.current.connection.clear_query_cache
  end