1

Are there any existing out of the box job queue framework? basic idea is

  1. someone to enqueue a job with job status New
  2. (multiple) workers get a job and work on it, mark the job as Taken. One job can only be running on at most one worker
  3. something will monitor the worker status, if the running jobs exceed predefined timeout, will be re-queued with status New, could be worker health issue
  4. Once a worker completes a task, it marks the task as Completed in the queue.
  5. something keeps cleaning up completed tasks. Or at step #4 when worker completes a task, the worker simply dequeues the task.

From my investigation, things like Kafka (pub/sub) or MQ (push/pull & pub/sub) or cache (Redis, Memcached) are mostly sufficient for this work. However, they all require some sort of development around its core functionality to become a fully functional job queue.

Also looked into relational DB, the ones supports "SELECT * FOR UPDATE SKIP LOCKED" syntax is also a good candidate, this again requires a daemon between the DB and worker, which means extra effort.

Also looked into the cloud solutions, Azure Queue storage, etc. similar assessment.

So my question is, is there any out of the box solution for job queue, that are tailored and dedicated for one thing, job queuing, without much effort to set up?

Thanks

dengying
  • 31
  • 1
  • 3
  • I think this depends on the type of jobs/tasks you are looking to run. Any details to provide there? – im_baby Jun 02 '21 at 02:05

1 Answers1

1

Take a look at Python Celery. https://docs.celeryproject.org/en/stable/getting-started/introduction.html

The default mode uses RabbitMQ as the message broker, but other options are available. Results can be stored in a DB if needed.

Chad Knutson
  • 351
  • 1
  • 4