How to setup docker compose with redis, sentinel and django+celery

Question

I'm having trouble with the setup of sentinel on docker. Everything I've found looks old and deprecated and having issues when I'm trying to run.

I'm using docker-compose with code like this :

redis-master
    image: redis:4.0.11-alpine
    volumes:
      - broker-data:/data
    networks:
      - db
    ports:
      - 6379:6379

  redis-slave:
    image: redis:4.0.11-alpine
    command: 
      - redis-server --slaveof redis-master 6379
    volumes:
      - broker-data:/data
    networks:
      - db
    links: 
      - redis-master

  sentinel:
    build: ./sentinel
    volumes: 
      - broker-data:/data
    networks: 
      - db
    ports:
      - 26379:26379
    links: 
      - redis-slave
      - redis-master

And sentinel from https://github.com/mustafaileri/redis-cluster-with-sentinel/tree/master/sentinel

This part seems to run correctly but Celery (onboarded in Django) doesn't found it.

CELERY_BROKER_URL="sentinel://sentinel:26379/0"

and I tried to change the transport options, but always have No master found for None error when starting django with docker-compose.

CELERY_BROKER_TRANSPORT_OPTIONS={"my_master":"redis-master", "master_name":"redis-master"}

Any idea about a mistake I made, or good tips to use sentinel on django with docker ?

Django 2.2.8
Celery 4.3.0

I bet the two Redis containers need separate backing data. (You can delete all of the `links:` and manual `networks:` setup without any significant impact to this setup.) — David Maze, Jan 03 '20 at 17:54

score 0 · Accepted Answer · answered Jan 20 '20 at 13:04

I've finally found the error. I used the sentinel for CELERY_RESULT_BACKEND and no option in the hidden CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS parameter was set.

In my case, here is the code in django to use celery and sentinel :

CELERY_BROKER_URL = "sentinel://sentinel1:26379/0;sentinel://sentinel2:26380/0"
CELERY_RESULT_BACKEND = "sentinel://sentinel1:26379/0;sentinel://sentinel2:26380/0"
CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": 3600, "master_name": "cluster1"}
CELERY_RESULT_BACKEND_TRANSPORT_OPTIONS = {"visibility_timeout": 3600, "master_name": "cluster1"}

How to setup docker compose with redis, sentinel and django+celery

1 Answers1