Howto know that I do not block Ruby eventmachine with a mongodb operation

Question

I am working on a eventmachine based application that periodically polls for changes of MongoDB stored documents.

A simplified code snippet could look like:

require 'rubygems'
require 'eventmachine'
require 'em-mongo'
require 'bson'

EM.run {

  @db       = EM::Mongo::Connection.new('localhost').db('foo_development')
  @posts    = @db.collection('posts')
  @comments = @db.collection('comments')

  def handle_changed_posts
    EM.next_tick do
      cursor = @posts.find(state: 'changed')
      resp = cursor.defer_as_a

      resp.callback do |documents|
        handle_comments documents.map{|h| h["comment_id"]}.map(&:to_s) unless documents.length == 0
      end

      resp.errback do |err|
        raise *err
      end
    end
  end

  def handle_comments comment_ids
    meta_product_ids.each do |id|
      cursor = @comments.find({_id: BSON::ObjectId(id)})
      resp = cursor.defer_as_a

      resp.callback do |documents|
        magic_value = documents.first['weight'].to_i * documents.first['importance'].to_i
      end

      resp.errback do |err|
        raise *err
      end
    end
  end

  EM.add_periodic_timer(1) do
    puts "alive: #{Time.now.to_i}"
  end

  EM.add_periodic_timer(5) do
    handle_changed_posts
  end
}

So every 5 seconds EM iterates over all posts, and selects the changed ones. For each changed post it stores the comment_id in an array. When done that array is passed to a handle_comments which loads every comment and does some calculation.

Now I have some difficulties in understanding:

1. I know, that this load_posts->load_comments->calculate cycle takes 3 seconds in a Rails console with 20000 posts, so it will not be much faster in EM. I schedule the handle_changed_posts method every 5 seconds which is fine unless the number of posts raises and the calculation takes longer than the 5 seconds after which the same run is scheduled again. In that case I'd have a problem soon. How to avoid that?

2. I trust em-mongo but I do not trust my EM knowledge. To monitor EM is still running I puts a timestamp every second. This seems to be working fine but gets a bit bumpy every 5 seconds when my calculation runs. Is that a sign, that I block the loop?

3. Is there any general way to find out if I block the loop?

4. Should I nice my eventmachine process with -19 to give it top OS prio always?

Answer 1

I have been reluctant to answer here since I've got no mongo experience so far, but considering no one is answering and some of the stuff here is general EM stuff I may be able to help:

1. schedule next scan on first scan's end (resp.callback and resp.errback in handle_changed_posts seem like good candidates to chain next scan), either with add_timer or with next_tick
2. probably, try handling your mongo trips more often so they handle smaller chunks of data, any cpu cycle hog inside your reactor would make your reactor loop too busy to accept events such as periodic timer ticks
3. no simple way, no. One idea would be to measure diff of Time.now to next_tick{Time.now}, do benchmark and then trace possible culprits when the diff crosses a threshold. Simulating slow queries (Simulate slow query in mongodb? ?) and many parallel connections is a good idea
4. I honestly don't know, I've never encountered people who do that, I expect it depends on other things running on that server

Answer 2

To expand upon bbozo's answer, specifically in relation to your second question, there is no time when you run code that you do not block the loop. In my experience, when we talk about 'non-blocking' code what we really mean is 'code that doesn't block very long'. Typically, these are very short periods of time (less than a millisecond), but they still block while executing.

Further, the only thing next_tick really does is to say 'do this, but not right now'. What you really want to do, as bbozo mentioned, is split up your processing over multiple ticks such that each iteration blocks for as little time as possible.

To use your own benchmarks, if 20,000 records takes about 3 seconds to process, 4,000 records should take about 0.6 seconds. This would be short enough to not usually affect your 1 second heartbeat. You could split it up even farther to reduce the amount of blockage and make the reactor run smoother, but it really depends on how much concurrency you need from the reactor.