Rails: how to debug localhost not responding with delayed_job / foreman

Question

I have a task that creates new activerecord records which I have recently moved to a background task using delayed_job and foreman, as recommended by Heroku

Sometimes this works fine, but sometimes it causes the Rails app in the browser to stop responding.

At this point I can see from the database that all the delayed jobs have completed, and that all the new records have been created.

However when I kill the processes I get a futher 11,200 lines of terminal output. This mostly consists of the execution by the web process of two methods on the model, both of which involve calls to the database:

validate :hit_database_to_see_if_model_exists?

before_save :get_rows_from_database_and_perform_calculation  

There are also a number of INSERT statements which I'm sure have already hit the database because the number of records does not change before/after killing the processes

Here is my Procfile:

web: bundle exec rails server thin -p $PORT -e $RACK_ENV

worker: bundle exec rake jobs:work

So it feels like I am getting a 'stack overflow' (woop). Can you shed any light on:

• What generally is going on?
• Where in Rails this 'stack overflow' is taking place
• Whether these things are actually happening after I hit 'Ctrl + C' or just being printed to the terminal at that point?
• What might be causing this?
• How I could debug / fix it?

UPDATE

It looks like there are certain tasks that are being assigned to the web process by the background task, but not being executed until the browser is 'prodded.' In some circumstances they all execute, but if there are too many the app falls over. Any idea on what might be causing this?

UPDATE

I tried running the web and worker processes in two separate windows.

In this scenario I have been unable to replicate the problem of the browser hanging, and in each case the worker process completed properly.

However I did make the interesting observation that if I don't touch the browser then no output appears in the web window. However if I do touch the browser then thousands of lines of what the worker process is doing at that moment appear in the web window.

Is this normal? Does this shed any light on what the problem might be?

Update

At the bottom of the terminal output after I kill the processes it says "Killed: 9"

07:45:21 system | sending SIGKILL to all processes

Killed: 9

What exactly does this 9 refer to? Is this unusual?

Update

I am using:

• delayed_job 3.0.4
• delayed_job_active_record 0.3.3
• delayed_job_web 1.1.2
• foreman 0.60.2

RESOLUTION

Thanks to @Justin's answer below (and this related question). It seems that Ruby buffers stdout by default, and that this buffer was overflowing, causing the app to stop responding. I added $stdout.sync = true at the top of config/environments/development.rb, and the problem appears to have gone away.

Answer 1

This is only a partial answer, but it might help you with debugging.

Rails buffers logging by default, and flushes it after every web request. One option is to simply replace the logger with a simpler logger

Rails.logger = Logger.new(STDOUT)

You can also configure the buffered logger to flush more often

Rails.logger.auto_flushing = (Rails.env.development? || Rails.env.test?)

You also have to be careful about STDOUT. In my current project I have stdout flushing enabled in both config.ru and my background bootup (I'm using sidekiq, so the boot process may be a little different)

STDOUT.sync = true

As to your larger problem,

I'm surprised that the rails process is running background tasks. Is there an option to disable that, at least for experimentation?

Then there are the standard debugging tools - all the database calls on save worry me, so I'd try various combinations of disabling them to see if anything improves. Especially with two of them - for instance, if your before_save hook changes a value in the model, it might be trigger a validation; if that resets the before_save hooks, you'd have a loop.