5

I have a rails app using delayed_job. I need my jobs to communicate with each other for things like "task 5 is done" or "this is the list of things that need to be processed for task 5".

Right now I have a special table just for this, and I always access the table inside a transaction. It's working fine. I want to build out a cleaner api/dsl for it, but first wanted to check if there were existing solutions for this already. Weirdly I haven't found a single things, I'm either googling completely wrong, or the task is so simple (set and get values inside a transaction) that no one has abstracted it out yet.

Am I missing something?

clarification: I'm not looking for a new queueing system, I'm looking for a way for background tasks to communicate with one another. Basically just safely shared variables. Do the below frameworks offer this facility? It's a shame that delayed job does not.

use case: "do these 5 tasks in parallel, and then when they are all done, do this 1 final task." So, each of the 5 tasks checks to see if it's the last one, and if it is, it fires off the final task.

John Bachir
  • 22,495
  • 29
  • 154
  • 227
  • Hm, can't see a real advantage of your approach unless you can specify a real case somebody need this. In my opinion I would keep the background workers as dumb as possible. So no shared variables. With one of the queueing system mentioned below you will be able to share data via the used databases (redis/memcache), with redis pub-sub you also can have communication between the workers depending on there subscriptions. There you can send your desired "task 5 is done" back to app or job manager or to other workers. Take a look into https://github.com/defunkt/resque "Resque vs DelayedJob". – asaaki Jun 11 '11 at 02:46
  • Resque/Redis is only my preference here, because you can easy combine the queueing and messaging with the same db system, that is the big advantage of it. In other queue systems you might have to implement the messaging completely by you own. If you want to stay on DelayedJobs you could implement the redis pub-sub stuff for messaging (or another messaging service) and it should be okay, too. No matter which final solution you take, I think messaging will be safer than a shared variable. – asaaki Jun 11 '11 at 02:59
  • Thanks for all the new info! I edited my question and described my use case. Does reque explicitly handle this sort of thing? – John Bachir Jun 13 '11 at 01:54
  • For your use case you could do a check on your different work queues, and don't really need messaging. Example: you have 2 queues, :taskA and :taskB, your app pushes 5 task into :taskA, 5 workers waiting for this queue and they catch each 1 task. After finishing the task each worker checks if the queue is empty now, if true the last worker will fire a task in :taskB queue. Alternative could be an watcher process which will do the checks. With pooling, namespaced resque jobs and much more cool stuff around resque you can do a lot of stuff mostly without real messaging. – asaaki Jun 13 '11 at 13:47

3 Answers3

3

I use resque. Also there are lots of plugins, which should make inter-process comms easier.

Using redis has another advantage: you can use the pub-sub channels for communication between workers/services.

Another approach (but untested by me): http://www.zeromq.org/, which also has ruby bindings. If you like to test new stuff, then try zeromq.

Update

To clarify/explain/extend my comments above:

Why I should switch from DelayedJobs to Resque is the mentioned advantage that I have queue and messages in one system because Redis offers this.

Further sources:

If I had to stay on DJ I would extend the worker classes with redis or zeromq/0mq (only examples here) to get the messaging in my extisting background jobs.

I would not try messaging with ActiveRecord/MySQL (not even queueing actually!) because this DB isn't the best performing system for this use case especially if the application has too many background workers and huge queues and uncountable message exchanges in short times.

If it is a small app with less workers you also could implement a simple messaging via DB, but also here I would prefer memcache instead; messages are short living data chunk which can be handled in-memory only.

Shared variables will never be a good solution. Think of multiple machines where your application and your workers can live on. How you would ensure a save variable transfer between them?

Okay, someone could mention DRb (distributed ruby) but it seems not really used anymore. (never seen a real world example so far)

If you want to play around with DRb however, read this short introduction.

My personal preference order: Messaging (real) > Database driven messaging > Variable sharing

asaaki
  • 1,970
  • 18
  • 22
2
DigitalRoss
  • 143,651
  • 25
  • 248
  • 329
0

You can use Pipes:

reader, writer = IO.pipe

fork do
  loop do
    payload = { name: 'Kris' }
    writer.puts Marshal.dump(payload)
    sleep(0.5)
  end
end

loop do
  begin
    Timeout::timeout(1) do
      puts Marshal.load(reader.gets) # => { name: 'Kris' }
    end
  rescue Timeout::Error
    # no-op, no messages to receive
  end
end
  • One way
  • Read as a byte stream

Pipes are expressed as a pair, a reader and a writer. To get two way communication you need two sets of pipes.

Kris
  • 19,188
  • 9
  • 91
  • 111