Ruby: EventMachine and separate callback queues

Question

EventMachine executes all operations in separate threads and all callbacks of this operations in one mainthread. It's very useful when you need reduce competitiveness in data processing between portions of data. You can recieve data from network concurent and process results in queue. But I need go even futher. I have many users on my site and want to make separate queue for each of them. So it should behave like single EM for each user. But I can't. EventMachine is singletone.

Thread.new do
  EM.run
end

def request_from_user
  operation = Proc.new {
    # network request
  }
  callback = Proc.new {
    # some heavy data processing
    2*2
  }
  5.times do
    EM.defer(operation, callback)
  end
end

3.times do
  request_from_user
end

So we have callback-mess in mainthread queue. Some data will be processed in end of queue, some in the middle and it's awful. Heavy load will enlarge the queue and some users will recieve results with very big delay. If I run request_from_user in single thread callbacks will be executed in one EM main thread anyway.

So what can I do with it? Is it possible to create separate callbacks queue for each user and execute it concurent or not? May be even without EventMachine. Celluloid, whatever...

I believe EM runs all operations in the "single" mainline thread, isn't that right? — rogerdpack, Mar 08 '13 at 00:49
Using EM.defer the operation will execute on a thread from EMs thread pool, the callback will execute on the main thread. — dj2, Oct 04 '13 at 03:11

score 1 · Answer 1 · answered Mar 08 '13 at 00:50

If you can listen on separate ports, then you could have one server/port per user. If not Ruby does have a Queue object that may be of some use to you: http://www.ruby-doc.org/stdlib-1.9.3/libdoc/thread/rdoc/Queue.html for farming processes out to other threads.

score 0 · Answer 2 · answered Oct 04 '13 at 03:15

Why not just move the work in callback either to operation (so, callback only fires when you're done processing the data), or to an EM.defer call? You don't want to be doing your heavy data processing on the main thread as that's going to cause all kinds of latency.

Ruby: EventMachine and separate callback queues

2 Answers2