Python: Distributed task queue for different specific workers

Question

I am looking for a python library / framework that manages task distribution (e.g. a task queue). However, tasks will require specialized workers: Worker A can only handle tasks of type a, workers B and C only of type b etc. Also, these workers will run on different computers and cannot share the same codebase (since, like in a fabrication line, each task is bound to controlling specific hardware, which only one computer has access to).

I have looked at libraries like python RQ or Celery, but if I understand correctly, they require the same codebase to run on the different workers and are meant for distributing computation. What I am looking for is basically just the management of an abstract task queue and a mechanism that workers can fetch tasks over the network. A task then is basically just some data and meta information about it's progress, errors, results etc. A bonus would be if tasks could also depend on one another, so that a task can depend on the outcome of another task.

Is there a simple library, that takes care of managing the queue, the network protocol etc., that provides what I'm looking for?

why not just use different queues for different kind of workloads? for type `a` have `queue-a` etc. you would be able to pass the data as well — Junaid, Jan 22 '19 at 12:50
You mean for RQ or Celery? Because as far as I understand, all workers need to have the same codebase and that's not ideal... — mrspl, Jan 23 '19 at 09:52
that is not true imho. you can have workers with differing codebases listening to different queues. — florian, Apr 23 '20 at 21:51

score 6 · Answer 1 · answered Feb 18 '19 at 00:21

This sounds like a great fit for Ray, which is a library for parallel and distributed Python. You can use Ray actors to create "specialized workers". Here's an example with multiple types of workers represented by classes WorkerType1 and WorkerType2.

import ray


@ray.remote
class WorkerType1(object):
    def method1(self):
        # Do some interesting work.
        return 1


@ray.remote
class WorkerType2(object):
    def method2(self):
        # Do some interesting work.
        return 2


if __name__ == "__main__":
    ray.init()

    # Create one worker of each type.
    worker_1 = WorkerType1.remote()
    worker_2 = WorkerType2.remote()

    # Kick off 100 tasks for each worker.
    result_ids = []
    for _ in range(100):
        result_ids.append(worker_1.method1.remote())
        result_ids.append(worker_2.method2.remote())

    # Retrieve the results.
    results = ray.get(result_ids)  # This is [1, 2, 1, 2, ...].

You can of course create a pool of workers of each type. The workers can also have state that the tasks mutate, can have multiple different methods, and so on. You can read more in the Ray documentation.

Note that to run the above example on a cluster instead of a single machine, you need to modify the ray.init() line to pass in a cluster address.

Note that I'm one of the Ray developers.

score 2 · Answer 2 · answered Jan 22 '19 at 13:05

I think that what you need is not a python library, but a proper queue service, that is configured and managed as a unit completely separate from the python workers. This way you do not need to write your own common libraries that would be the common dependency of all the workers, but can reuse already existing packages for queue subscription.

I really recommend this resource to aid you with what you want to achieve and introduce terminology to look for: https://github.com/donnemartin/system-design-primer#message-queues

I linked into message queues section, that lists several options you can explore:

Redis - this is actually what you proposed - note that Redis is not persistent and your unfinished tasks can be lost.
RabbitMQ - I personally recommend this one as it uses AMQP protocol and has a vast community.
Amazon SQS - If you use AWS, this service has the upside of being managed in a PaaS model.

Off course, queues can be used to distribute tasks as messages.

Additionally, if you are building a highly asynchronous system, you could explore Event Sourcing pattern, as it shapes your whole architecture to use messages queues or streaming services to propagate events. If you want to go down this road, then picking right service for the job is very important.

Yes, I have looked at those message brokers, but I was hoping for something just a little more high level - so basically something that already implements the terminology for jobs/tasks and provides info such as job status, progress, errors etc. But maybe I'll just have to implement it myself on top of such a message broker... — mrspl, Jan 23 '19 at 09:55

Python: Distributed task queue for different specific workers

2 Answers2