2

I am looking to run queued jobs using RQ, but looking at the following example:

from rq import Queue
from redis import Redis
from somewhere import count_words_at_url

# Tell RQ what Redis connection to use
redis_conn = Redis()
q = Queue(connection=redis_conn)  # no args implies the default queue

# Delay execution of count_words_at_url('http://nvie.com')
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job.result   # => None

# Now, wait a while, until the worker is finished
time.sleep(2)
print job.result   # => 889

I see time.sleep(2) - and I was wondering if this is mandatory to specify. The jobs I schedule could take (at times) an hour to complete (this varies on a per job basis).

Is RQ still suitable for such jobs - where execution time vastly varies?

Any suggestions would be great!

AJW
  • 5,569
  • 10
  • 44
  • 57

1 Answers1

4

As long as each job you schedule gives some clear, observable indication that it's all done, you can definitely use RQ and wait for such "indications"; then you'll rely on such indications to be able to tell when to access each job's result.

In the example you quote, it's apparently assumed that count_words_at_url gives no clear indication about when it's finished (you could "poll" by looping on while job.result is None: with a sleep in the while's body, but that's pretty fragile -- avoid polling when you can).

Alex Martelli
  • 854,459
  • 170
  • 1,222
  • 1,395
  • I am sorry to ask, but, could you point me to an example of how this (waiting for `indications`) is done on a production environment? Also, what sort of `indications` are we talking about here? like a bash exit code? – AJW Dec 23 '14 at 18:30
  • A typical example would be to have each `heavy` job write a file to a specified directory and with a specified name as it finishes. Then your "master" watches for events in that directory, e.g with `watchdog` from https://pypi.python.org/pypi/watchdog , and reacts to them appropriately . Files are just one example of a way for processes to communicate without "polling", but still OK for many uses as long as you're working within a single system; for fully distributed systems more advanced approaches are needed. – Alex Martelli Dec 23 '14 at 18:39
  • Thanks for the `watchdog` tip - I will definitely use this. So, first I let a `heavy job` write a file out when the job is fully done (say `done.txt`) and let watchdog monitor this - how do I integrate this with RQ? i.e how do I let RQ know that the job is done? (just looking for `done.txt` within RQ does not make sense..) Sorry, think I am a bit lost here. – AJW Dec 23 '14 at 18:53
  • Ah.. I think I see it.. So I just let `watchdog` monitor for `done.txt` and if present then `print job.result` else not. Is that the gist of it? – AJW Dec 23 '14 at 18:57
  • Yes, except that you'd typically have many jobs so you'd be looking for differently-named files from each of them. – Alex Martelli Dec 23 '14 at 20:01
  • Fair point Alex - thanks so much for all your help and advice. I will implement your suggestion - looks clean. Accepted your answer :) – AJW Dec 23 '14 at 20:17