7

I have multiple applications written with nodejs or python/django or ... These services are working fine. But need to have pub/sub Async communication with each other.

In nodejs there is no problem and easily can pub/sub to any redis channel.

Question: My question is how can i continuously subscribe to a redis channel and receive data published with other services?

Note: many links suggest to use django-channels. But I guess that's not the way to do it. If so can any one help me and give details on how to do it.

Update: Django by default is not event-based like nodejs. So if i use a redis client, I should for example check redis every second and see if anything is published or not. I don't think using just a redis client in python will be enough.

Really appreciate it.

Reza Torkaman Ahmadi
  • 2,958
  • 2
  • 20
  • 43
  • 1
    Why don’t you use Redis Pub/Sub in python django. I mean, why you can’t use Redis in django? Subscribe to channel from NodeJS? – Gor Kotikyan Jan 31 '19 at 16:33
  • 1
    I haven't worked it yet. But using `django-celery`[link](http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html) can be a good option for async pub/subs task. – Shakil Jan 31 '19 at 16:34
  • The problem is django is not event-based. If i want to have nodejs like performance, I should use real-time frameworks of django. like twisted. – Reza Torkaman Ahmadi Jan 31 '19 at 17:37
  • And django-celery I did used. But it's more of a task-queue. I don't think that's the exact solution for my problem. – Reza Torkaman Ahmadi Jan 31 '19 at 17:38

2 Answers2

8

There are a lot of alternatives. If you have FIFO issue you have to use queues in order to connect one microservice to another. For me, if you don’t have Big Data problem you can use RabbitMQ, It is very practical and very effective, otherwise if you have Big Data problem you can use Kafka. There are wide variety services.

If you want just Pub/Sub. The best tool is Redis, It is very fast and easy to integrate. If you are concerned how to implement it in Python just look at article

[Update]

It's possible to create a manage.py command in django and subscribe to redis in that management file and execute this script separated from django server:

class Command(BaseCommand):
def handle(self, *args, **options):

    r = redis.StrictRedis(host='localhost', port=6379, db=1)
    p = r.pubsub()
    p.psubscribe('topic.*')
    for message in p.listen():
        if message:
            print('message received, do anything you want with it.')
Community
  • 1
  • 1
Gor Kotikyan
  • 723
  • 4
  • 12
  • So you are saying that I should run a python script beside my django application that always listen to redis. But i miss the ORM features of django. What if i create a `manage.py` script in django and look for anything that published on redis? – Reza Torkaman Ahmadi Jan 31 '19 at 17:33
  • It is also an option. I assume you also can use [Django Redis Pub/Sub](https://pypi.org/project/django-redis-pubsub/) library. – Gor Kotikyan Jan 31 '19 at 21:38
  • No it's not what I want. I just want a simple pub/sub solution with redis for django. – Reza Torkaman Ahmadi Feb 01 '19 at 05:18
  • And what happens if you want to deploy this in production? – pac Oct 21 '22 at 22:47
6

In order to handle subscriptions to redis, you will need to have a separate continuously running process (server) which listens to redis and then makes something with your data. django-channels will do the same by running the code in a worker

As pointed above, Django provides convenient way to run "servers" by using Django management command approach. When running a django management command, you have complete access to your code, i.e. to ORM as well.

Only detail that you mentioned Async communication. Here you need to take into account that Django's ORM is strictly sync code, and you need to pay attention how you want to use ORM with async code. Probably you need to clarify what do you mean by async here.

As for redis messages processing, you can use any libraries that work with it. For example, aioredis or redis-py

gvm
  • 131
  • 1
  • 3