3

I would like to implement a queue in redis + Flask and Python. I already implemented a query like this with RQ and it works fine if you have the Flask app and the task working on the same server. I am wondering if it is possible to create a queue (multi-consumer) where the worker(s) are actually on an other server. For example:

Client post data to Flask -> Flask create an item in the Redis queue -> Redis queue is picked up by some workers on an other server (backend).

Since the code in Flask is something like:

redis_conn = Redis()
q = Queue('my_queue', connection=redis_conn)
job = q.enqueue_call(func='myqueue.myfunc', args=(json,), result_ttl=5000)

Obviously 'myqueue.myfunc' needs to stay on the Flask server but I would like to be able to push the data and have a worker on another server. Do you know if this is feasible or what are other things that can be used to solve this problem?

Thanks.

Rob
  • 89
  • 6
  • And by the way, I thought about Celery, but I wonder if there other possible solutions. – Rob Nov 30 '19 at 20:06
  • I see no one problem to connect many workers to your redis server. Can you describe you problem clearly? – Peter Dec 01 '19 at 10:38
  • Also there are lots of different solution of distributed queue. Just a quick googling shows: https://dramatiq.io/, http://python-rq.org/, https://www.rabbitmq.com/ – Peter Dec 01 '19 at 10:41
  • Hi Peter, what I am trying to do is building a redis queue (I want to use Redis and no other technologies like RabbitMQ - although it is actually good). Basically I want to have the possibility to push data into Redis (JSON data) and have multiple workers (on multiple servers) parsing the data, perform some heavy memory intensive tasks and then save the data into the DB. I would like to have for example 2/3 servers in the backend (not accessible from Public IP) and have 5/10 workers on each server to handle the Redis queue. I thought to use "rpush", but I wonder if there are better solutions. – Rob Dec 01 '19 at 15:41
  • I am not sure how to implement properly Python-RQ on a distributed environment as from Flask when I do the enqueue I need to provide the class name/function name and the class name/function name is on another server. – Rob Dec 01 '19 at 15:44
  • I have solved it. – Rob Dec 07 '19 at 14:15
  • How did you solve it?! post the answer please – Neil Jan 05 '22 at 10:44
  • @Rob Please post an answer to your own question if you solved it. – Cow Jan 11 '22 at 09:47

1 Answers1

3

The pattern of having queue workers on other nodes is common.

So in your flask app node configure a remote redis server

r = redis.Redis(host='myredis.example.com', port=6379)

Then on one or more worker nodes

from redis import Redis
from rq import Queue, Worker

# Returns all workers registered in this connection
redis = Redis(host='myredis.example.com', port=6379)
workers = Worker.all(connection=redis)

# Returns all workers in this queue (new in version 0.10.0)
queue = Queue('queue_name')
workers = Worker.all(queue=queue)
worker = workers[0]
print(worker.name)
namizaru
  • 646
  • 3
  • 5