12

How does the rpc result backend differ from the amqp backend? I see in the changelog that it replaced it, but although it is written as a protocol (with the ://), the underlying protocol is still amqp, correct?

For instance, result_backend = 'rpc://' vs result_backend = 'amqp://'. If I use rpc as the backend, does it also use SSL when the broker_use_ssl flag is set to true?

john
  • 3,043
  • 5
  • 27
  • 48
  • 3
    celery documentation isn't clear on this. I remember I came across somewhere which talks about using amqp as backend may introduce performance overhead. I can't remember the link anymore. I add a bounty hoping someone with insider knowledge may explain different between rpc and amqp, and how to choose among them. – Cheok Yan Cheng Feb 21 '18 at 14:56

1 Answers1

31

Consider a scenario, where 4 clients have to queue 100 tasks each.

In case of amqp backend, it will create 400 unique queues and stores results in those queues.

In case of rpc backend, it will create only 4 queues(1 per client) and stores 100 results in each queue which results in significant improvement in performance as there is no overhead to create queues for each and every task.

For this reason, amqp as backend is deprecated and will be completely removed in next release.

rpc backend uses the same publish/consume mechanism of amqp. If you set broker_use_ssl to True, then it will use SSL.

Chillar Anand
  • 27,936
  • 9
  • 119
  • 136
  • Do you mind to further explain what is SSL in celery context? Is it same as "Secure Sockets Layer"? How should we decide whether to enable `broker_use_ssl` or not? – Cheok Yan Cheng Feb 22 '18 at 17:22
  • @CheokYanCheng Yes, it is the same. Whether to enable or not depends on how your client and server are connecting, if the transferring data is confidential or not, or if you need to follow some standards like HIPAA etc. – Chillar Anand Feb 23 '18 at 08:57
  • Can you explain if their result persistent behaviors are different. I run into the a similar problem as https://stackoverflow.com/questions/34541282/celery-task-results-not-persisted-with-rpc , where I can't enable result_persistent using rpc. – Jzou Jul 05 '18 at 20:21
  • @JialinZou Are you getting any error while enabling it? If result_persistent is enabled, it will persist http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-result_persistent – Chillar Anand Jul 06 '18 at 09:23
  • @ChillarAnand No error, I can enable result_persistent in both amqp and rpc. The difference is that when I use amqp, I can retrieve result in another Python interpreter by AsyncResult(task_id). However using rpc, this only works within the same interpreter. Not sure this is a bug or designed to be. I didn't find answer in Celery documents. – Jzou Jul 06 '18 at 15:13
  • Are chords going to be supported by the RPC backend? It seemed to lack features, I also had trouble getting back the status of chained tasks with RPC backend. – KillerKode Sep 29 '20 at 10:15