Celery Cannot connect to sentinel: No master found for 'mymaster'

Question

I have a working redis sentinel. I can connect to it with python code and ping it without issue (code below). However when I try to connect my celery application to this redis sentinel, I get back no master found for 'mymaster'. I probably have some settings wrong? both my master and my sentinel are protected by password. I am running it in Openshift, though I am not sure it is relevant as the sentinel is accessible by the python code from a different pod (and service).

redis-cli -p 26379 -h 192.168.7.248
192.168.7.248:26379> auth abc
OK
192.168.7.248:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.18.36:6379,slaves=0,sentinels=1

import redis
from redis.sentinel import Sentinel

sentinel = Sentinel([('192.168.7.248', 26379)], sentinel_kwargs={'password': 'abc'})
host, port = sentinel.discover_master("mymaster")
redis_client = redis.StrictRedis(
        host=host,
        port=port,
        password='abc'
    )
print(redis_client.ping())

This code prints out "True".

My celery app:

from celery import Celery

app = Celery("app_celery")

options = {
    "master_name": "mymaster",
    "sentinel_kwargs": {"password": "abc"},
}
app.conf.broker_url = "sentinel://192.168.7.248:26379"
app.conf.broker_transport_options = options

I am absolutely desperate, any help would be really appreciated.

score 3 · Answer 1 · answered Apr 21 '23 at 20:14

Celery documentation shows how to configure your sentinel password only. https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html

If both your master and sentinel are password protected then below is how you can initialize your Celery app.

from celery import Celery

REDIS_PASSWORD = 'your_redis_password'
REDIS_SENTINEL_PASSWORD = 'your_redis_sentinel_password'

redis_broker = f'sentinel://:{REDIS_PASSWORD}@localhost:26379;sentinel://:{REDIS_PASSWORD}@localhost:26380;sentinel://:{REDIS_PASSWORD}@localhost:26381'

app = Celery('app_celery', broker=redis_broker)

app.conf.broker_transport_options = {'master_name': 'mymaster', 'sentinel_kwargs': { 'password': REDIS_SENTINEL_PASSWORD }}

ovimunt · Answer 2 · 2022-03-30T23:23:14.340

It's nowhere in the docs, but this is what worked for me:

broker_url = "sentinel://:your_password@host_ip:port_no;..."
broker_transport_options = {"master_name": "your_master_name"}

So include the password inside the broker_url for each of the sentinel hosts.

Further details

Celery uses Kombu under the hood. Kombu actually gives you the option to not include the password under the transport_options, which Celery does not translate across. So in Kombu you can do this:

c = kombu.Connection(
        "sentinel://host_ip:port_no;...",
        "password": "your_password",
        transport_options={"master_name": "your_master_name"},
    )
c.connect()

If your connection and sentinel hosts password is the same, this works perfect.

But what Celery does is to stick the password either inside the transport_options or all the way inside the sentinel_kwargs. So effectively this is what gets sent to Kombu.

celery_broker_transport_options = {
    "master_name": "your_master_name",
    "password": "your_password"
}

# gets into Kombu like this
c = kombu.Connection(
        "sentinel://host_ip:port_no;...",
        transport_options={
            "master_name": "your_master_name",
            "password": "your_password",
        }
    )


# OR

celery_broker_transport_options = {
    "master_name": "your_master_name",
    "sentinel_kwargs": {"password": "your_password"}
}

# gets into Kombu like this
c = kombu.Connection(
        "sentinel://host_ip:port_no;...",
        transport_options={
            "master_name": "your_master_name",
            "sentinel_kwargs": {"password": "your_password"}
        }
    )

Neither one of these cases work.

So, by including the sentinel host password inside the broker_url you are getting around this... bug I would call it.

This answer was helpful knowing that what might sound intuitive isn't what might actually happen and the error message is misleading. In my case, I needed to pass my redis password to `sentinel_kwargs` in addition to the password in the broker URL. Instead of getting an authentication error, I would get No master found. — Bufke, Jul 22 '22 at 15:35

Celery Cannot connect to sentinel: No master found for 'mymaster'

2 Answers2

Further details