2

We are using Flower to monitor Celery tasks with Redis as a broker in our project.

When we configured Redis with SSL as a broker, in the broker tab we can see the next message:

'redis' broker is not supported

We tried to (and successfully) connect to Redis from Celery through the next URL:

rediss://{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB_NAME}?{SSL_OPTIONS}

as indicates in Celery docs.

Also, we tried both --broker and --broker_api parameters with the same result.

The workers and tasks work fine (and inside the worker, the brokers appear as normally) but the broker tab is empty.

If we look at the Celery code, we can find how Celery manage the url parsing to set the ssl options and configuration for Redis as a broker:

        if scheme == 'redis':
            # If connparams or query string contain ssl params, raise error
            if (any(key in connparams for key in ssl_param_keys) or
                    any(key in query for key in ssl_param_keys)):
                raise ValueError(E_REDIS_SSL_PARAMS_AND_SCHEME_MISMATCH)

        if scheme == 'rediss':
            connparams['connection_class'] = redis.SSLConnection
            # The following parameters, if present in the URL, are encoded. We
            # must add the decoded values to connparams.
            for ssl_setting in ssl_param_keys:
                ssl_val = query.pop(ssl_setting, None)
                if ssl_val:
                    connparams[ssl_setting] = unquote(ssl_val)

And if we look at the Flower code, we can find how Flower manage the url parsing to set the broker:

        class Broker(object):
            def __new__(cls, broker_url, *args, **kwargs):
                 scheme = urlparse(broker_url).scheme
                 if scheme == 'amqp':
                     return RabbitMQ(broker_url, *args, **kwargs)
                 elif scheme == 'redis':
                     return Redis(broker_url, *args, **kwargs)
                 elif scheme == 'redis+socket':
                     return RedisSocket(broker_url, *args, **kwargs)
                 else:
                     raise NotImplementedError

As we can see, it seems that Flower doesn't support Redis with SSL (only support redis, not rediss).

Is this true? Is there any way to connect with Redis using rediss in both Celery and Flower or to connect with Redis using SSL without rediss?

Thanks!

Nuno André
  • 4,739
  • 1
  • 33
  • 46
Pablo G
  • 21
  • 3

1 Answers1

0

It works with below URL options.
Both percent-encoded or normal paths for certs work.
Both rediss and redis schemes work.
Using Celery==4.4.7, flower==0.9.5

export SSL_CACERT_PATH="%2Fvar%2Fssl%2Fmyca.pem"
export SSL_CLIENT_CERT_PATH="%2Fvar%2Fssl%2Fclient-cert.pem"
export SSL_CLIENT_KEY_PATH="%2Fvar%2Fssl%2Fprivate%2Fworker-key.pem"
OR 
export SSL_CACERT_PATH=/var/ssl/myca.pem
export SSL_CLIENT_CERT_PATH=/var/ssl/client-cert.pem
export SSL_CLIENT_KEY_PATH=/var/ssl/private/worker-key.pem

#Ensure no whitespaces if breaking lines
export REDISS_BROKER_URL="rediss://:${REDIS_PASSWORD}@${REDIS_HOST}:${REDIS_PORT}?\
ssl_cert_reqs=required\
&ssl_ca_certs=${SSL_CACERT_PATH}\
&ssl_certfile=${SSL_CLIENT_CERT_PATH}\
&ssl_keyfile=${SSL_CLIENT_KEY_PATH}"

flower -b $REDISS_BROKER_URL

#Ensure no whitespaces if breaking lines
export REDIS_BROKER_URL="redis://:${REDIS_PASSWORD}@${REDIS_HOST}:${REDIS_PORT}?\
ssl_cert_reqs=required\
&ssl_ca_certs=${SSL_CACERT_PATH}\
&ssl_certfile=${SSL_CLIENT_CERT_PATH}\
&ssl_keyfile=${SSL_CLIENT_KEY_PATH}"

flower -b $REDIS_BROKER_URL
[I 201023 07:20:09 command:140] Visit me at http://localhost:5555
[I 201023 07:20:09 command:145] Broker: redis://:**@xxxx:xxxx//
[I 201023 07:20:09 command:148] Registered tasks:
    ['celery.accumulate',
     'celery.backend_cleanup',
     'celery.chain',
     'celery.chord',
     'celery.chord_unlock',
     'celery.chunks',
     'celery.group',
     'celery.map',
     'celery.starmap']
[I 201023 07:20:10 mixins:229] Connected to redis://:**@xxxx:xxxx//
^C^C
Kunal
  • 126
  • 1
  • 8