Celery Flower returning 500s on Broker Stats with Redis and SSL

Question

I just installed celery flower.

It is working great for showing me real-time tasks, which queue they were processed on, cpu usage, and the processing time.

I also want to have access to the broker page so I can monitor queue lengths.

The issue I am having is with SSL.

The broker page returns a 500. Looking at the logs, I am seeing the following stack trace.

2020-12-24T21:19:21.828079+00:00 app[web.1]: [W 201224 21:19:21 connection:255] Secure redis scheme specified (rediss) with no ssl options, defaulting to insecure SSL behaviour.
2020-12-24T21:19:21.854471+00:00 app[web.1]: [W 201224 21:19:21 connection:255] Secure redis scheme specified (rediss) with no ssl options, defaulting to insecure SSL behaviour.
2020-12-24T21:19:21.878474+00:00 app[web.1]: [E 201224 21:19:21 web:1793] Uncaught exception GET /broker (...)
2020-12-24T21:19:21.878479+00:00 app[web.1]: HTTPServerRequest(protocol='http', host='...herokuapp.com', method='GET', uri='/broker', version='HTTP/1.1', remote_ip='...')
2020-12-24T21:19:21.878480+00:00 app[web.1]: Traceback (most recent call last):
2020-12-24T21:19:21.878481+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tornado/web.py", line 1704, in _execute
2020-12-24T21:19:21.878481+00:00 app[web.1]: result = await result
2020-12-24T21:19:21.878482+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tornado/gen.py", line 234, in wrapper
2020-12-24T21:19:21.878482+00:00 app[web.1]: yielded = ctx_run(next, result)
2020-12-24T21:19:21.878482+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/tornado/gen.py", line 162, in _fake_ctx_run
2020-12-24T21:19:21.878483+00:00 app[web.1]: return f(*args, **kw)
2020-12-24T21:19:21.878483+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flower/views/broker.py", line 31, in get
2020-12-24T21:19:21.878485+00:00 app[web.1]: http_api=http_api, broker_options=broker_options, broker_use_ssl=broker_use_ssl)
2020-12-24T21:19:21.878485+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flower/utils/broker.py", line 237, in __new__
2020-12-24T21:19:21.878486+00:00 app[web.1]: return RedisSsl(broker_url, *args, **kwargs)
2020-12-24T21:19:21.878486+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flower/utils/broker.py", line 220, in __init__
2020-12-24T21:19:21.878486+00:00 app[web.1]: super(RedisSsl, self).__init__(broker_url, *args, **kwargs)
2020-12-24T21:19:21.878487+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flower/utils/broker.py", line 134, in __init__
2020-12-24T21:19:21.878487+00:00 app[web.1]: self.redis = self._get_redis_client()
2020-12-24T21:19:21.878488+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flower/utils/broker.py", line 155, in _get_redis_client
2020-12-24T21:19:21.878488+00:00 app[web.1]: return redis.Redis(**self._get_redis_client_args())
2020-12-24T21:19:21.878489+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/flower/utils/broker.py", line 225, in _get_redis_client_args
2020-12-24T21:19:21.878489+00:00 app[web.1]: client_args.update(self.broker_use_ssl)
2020-12-24T21:19:21.878554+00:00 app[web.1]: TypeError: 'NoneType' object is not iterable
2020-12-24T21:19:21.881667+00:00 app[web.1]: [W 201224 21:19:21 connection:255] Secure redis scheme specified (rediss) with no ssl options, defaulting to insecure SSL behaviour.

It looks like I need to pass in the cert somehow to broker_use_ssl, but I am not sure where or how.

This is all deployed on Heroku. The rediss URL is what's on my production application, flower is on a separate app.

On celery I have {"ssl_cert_reqs": ssl.CERT_NONE}.

Flower deployed on heroku looks like

requirements.txt as follows

celery==4.4.4
future==0.18.2
flower==0.9.7 
redis==3.5.3

Then a Procfile where I try to pass in ssl.CERT_NONE which returns 0. It doesn't work.

web: flower --port=$PORT --broker=$BROKER_URL --basic_auth=$FLOWER_BASIC_AUTH --broker_use_ssl={"ssl_cert_reqs": 0}

Can anyone shed some light on how to setup these configuration options?

Thank you

Answer 1

Seems like a fix related to broker_use_ssl was merged to master two days ago. Not sure if it only improves or bug fixes. There's a related issue here. Note that the latest release doesn't contain this fix yet (released 4 days ago).

Anyway, here is things that you can try:

1. Something with the way you're passing the value of --broker_use_ssl - maybe you need to escape the quotes, something like: --broker_use_ssl={\"ssl_cert_reqs\": 0} or --broker_use_ssl="{\"ssl_cert_reqs\": 0}".
2. Try to pass your settings via a configuration file instead of the command line, like: flower --conf=celeryconfig.py - that way you don't need to handle the escaping and you can set the value as you did ({"ssl_cert_reqs": 0}).
3. Use the master branch to see if the last commit solves your problem.

Good luck!