3

I am trying to setup an airflow cluster. I am planning to use redis as celery backend. I have seen people using sentinel redis successfully. I wanted to know if it is possible to use redis cluster instead? If not then why not?

Vishal Khare
  • 41
  • 1
  • 5

1 Answers1

2

Celery doesn't have support for using Redis cluster as broker. It can use Redis highly available setup as broker (with Sentinels), but has no support for Redis cluster to be used as broker.

Reference:

Airflow CROSSSLOT Keys in request don't hash to the same slot error using AWS ElastiCache

How to use more than 2 redis nodes in django celery

To make Redis cluster to work we need to change the celery backend! not a feasible solution.

https://github.com/hbasria/celery-redis-cluster-backend

Sathish Kumar
  • 43
  • 6