2

There are three instances, each of which is running a Django application with a celery queue on a redis broker and backend.

On a fourth instance, we are trying to set up a Flower monitor to keep tabs on these first three instances. There should be a Flower monitor on each of three different ports.

All four instances and the redis brokers are in Amazon Web Services, so permissions could be an issue. We've experimented opening all traffic between the security groups for the instances and the redis instances, but to no avail.

All three flower processes dutifully stand up and the Flower webpages are totally functional. The problem is that absolutely no content is coming in: no queue in the broker tab, no workers in the worker tab, and no tasks in the task tab. I didn't notice anything amiss in the logs, but I am open to suggestions for potential checks.

Currently each of the first three instances has their own flower that works just fine with no problems. The goal is to collect the Flowers onto a single instance to streamline monitoring.

What could be going wrong? How can I diagnose this?

PS: I can redis-cli into all the redis instances from the sentinel node.

I've tried two command patterns:

pathto/bin/celery --port=5555 --broker=redis://ourbroker:xxxx//

This didn't work, so I built up a Celery object with broker and backend arguments and tried this:

pathto/bin/celery --port=5555 -A app_name flower

Both of these produced the symptoms described.

rschwieb
  • 746
  • 2
  • 16
  • 41
  • Could you show the configuration of each celery worker, and also the command to start the flower – Jacky1205 Aug 05 '15 at 13:10
  • @Jacky Can you tell me (or point me to the docs) about where the worker configuration information is? – rschwieb Aug 05 '15 at 15:14
  • Take a look at http://celery.readthedocs.org/en/latest/configuration.html The important ones are `BROKER_URL`, `CELERY_IMPORTS`, `CELERY_RESULT_BACKEND` – Jacky1205 Aug 06 '15 at 01:09
  • @Jacky added the commands used and the arguments supplied. As far as I know, the configuration you describe is just fine on the worker nodes (it works perfectly in the native flowers) but none of that configuration is on the sentinel node. Therefore, I am guessing the configuration needs to be provided to the flower command. I've asked about this before but gotten nowhere. – rschwieb Aug 06 '15 at 15:10
  • @Jacky Any ideas occur to you? – rschwieb Aug 10 '15 at 14:47
  • @Jacky I've fallen back to the --broker --port version because it looks like the -A version requires all the code for tasks which is unacceptable. The right broker address appears in the monitor, but no workers are visible. – rschwieb Aug 24 '15 at 19:11
  • By now Flower does not support redis sentinel; You could take a look at https://github.com/mher/flower/blob/master/flower/utils/broker.py#L102 to see how Flower dealing with the passed broker address. The proper solution is to modify Flower to support sentinel node. – Jacky1205 Aug 25 '15 at 04:01

0 Answers0