Django + Celery + Flower seems to break RabbitMQ broker

Question

I've working on a fresh headless Debian server (Buster) with a still in dev setup of Django (3.1.6) + Celery (4.4.7) + RabbitMQ (3.7.8) + Flower (0.9.7). (I was running with Celery 5, but downgraded to 4.4.7, because of a bug.)

Django, Celery, and Flower are Pip installed and RabbitMQ by Debian apt-get

The code is from demo examples of Celery First Steps with Django (with names changed to spare the innocent).

I'm able to get a basic Celery example to work (see first example below). I'm also monitoring this headless server on another machine with Django/admin and RabbitMQ Management Console using ip:port. I'm trying to add Flower to this as well, but I'm encountering startup error as seen in the second Celery startup example below.

Django Site: http://192.168.1.102:8000/admin/django_celery_beat/
RabbitMQ Management Console: http://192.168.1.102:15672/
Flower: http://192.168.1.102:5555/dashboard

Starting Celery without flower argument:

> celery -A pyauto beat -l INFO
celery beat v4.4.7 (cliffs) is starting.
__    -    ... __   -        _
LocalTime -> 2021-02-13 19:55:19
Configuration ->
    . broker -> amqp://<user>:**@localhost:5672/<vhost>
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> django_celery_beat.schedulers.DatabaseScheduler

    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 seconds (5s)
[2021-02-13 19:55:19,862: INFO/MainProcess] beat: Starting...
[2021-02-13 19:55:34,883: INFO/MainProcess] Scheduler: Sending due task Hello World (<myapp>.tasks.hello_world)
[2021-02-13 19:55:49,874: INFO/MainProcess] Scheduler: Sending due task Hello World (<myapp>.tasks.hello_world)
[...]

Starting Celery with `flower` argument:

> celery flower -A pyauto beat -l INFO 
[I 210213 20:59:53 command:135] Visit me at http://localhost:5555
[I 210213 20:59:53 command:142] Broker: amqp://<user>:**@localhost:5672/<vhost>
[I 210213 20:59:53 command:143] Registered tasks:
    ['celery.accumulate',
     'celery.backend_cleanup',
     'celery.chain',
     'celery.chord',
     'celery.chord_unlock',
     'celery.chunks',
     'celery.group',
     'celery.map',
     'celery.starmap',
     'mailgoblin.tasks.add',
     'mailgoblin.tasks.count_widgets',
     'mailgoblin.tasks.hello_world',
     'mailgoblin.tasks.mul',
     'mailgoblin.tasks.rename_widget',
     'mailgoblin.tasks.xsum',
     'pyauto.celery.debug_task']
[I 210213 20:59:53 mixins:229] Connected to amqp://<user>:**@127.0.0.1:5672/<vhost>
[W 210213 20:59:54 inspector:42] Inspect method stats failed
[W 210213 20:59:54 inspector:42] Inspect method active_queues failed
[W 210213 20:59:54 inspector:42] Inspect method active failed
[W 210213 20:59:54 inspector:42] Inspect method reserved failed
[W 210213 20:59:54 inspector:42] Inspect method revoked failed
[W 210213 20:59:54 inspector:42] Inspect method conf failed
[W 210213 20:59:54 inspector:42] Inspect method scheduled failed
[W 210213 20:59:54 inspector:42] Inspect method registered failed
[E 210213 19:56:51 broker:78] RabbitMQ management API call failed: [Errno 104] Connection reset by peer

To diagnose the problem I've tried:

restart rabbitmq

Celery github issues describe a few things

$ celery -A pyauto inspect ping
Error: No nodes replied within time constraint.

I don't quite understand this bit of advice from Arch wiki. Check if your default user for the service can access erlang. Doesn't quite work as expected (and it's working anyway because of the first Celery example above):
```
$ su rabbitmq -c 'erl'
This account is currently not available.
```
another Celery issues recommended diagnostic, Celery ... report:

> celery -A pyauto report
software -> celery:4.4.7 (cliffs) kombu:4.6.11 py:3.8.2
            billiard:3.6.3.0 py-amqp:2.6.1
platform -> system:Linux arch:64bit, ELF
            kernel version:4.19.0-13-amd64 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:django-db

ABSOLUTE_URL_OVERRIDES: {
 }
ADMINS: []
ALLOWED_HOSTS: ['192.168.1.101',
 '192.168.1.102',
 '127.0.0.1',
 'localhost']
APPEND_SLASH: True
AUTHENTICATION_BACKENDS: ['django.contrib.auth.backends.ModelBackend']
AUTH_PASSWORD_VALIDATORS: '********'
AUTH_USER_MODEL: 'users.CustomUser'
BASE_DIR: PosixPath('/home/.../pyauto/src')
CACHES: {
 'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}
CACHE_MIDDLEWARE_ALIAS: 'default'
CACHE_MIDDLEWARE_KEY_PREFIX: '********'
CACHE_MIDDLEWARE_SECONDS: 600
CELERY_ACCEPT_CONTENT: ['json']
CELERY_BEAT_SCHEDULER: 'django_celery_beat.schedulers:DatabaseScheduler'
CELERY_BROKER_URL: 'amqp://<user>:********@localhost:5672/<vhost>'
CELERY_RESULT_BACKEND: 'django-db'
CELERY_TASK_SERIALIZER: 'json'
CELERY_TASK_TIME_LIMIT: 1800
CELERY_TASK_TRACK_STARTED: True
CELERY_TIMEZONE: 'America/New_York'
CRISPY_TEMPLATE_PACK: 'bootstrap4'
CSRF_COOKIE_AGE: 31449600
CSRF_COOKIE_DOMAIN: None
CSRF_COOKIE_HTTPONLY: False
CSRF_COOKIE_NAME: 'csrftoken'
CSRF_COOKIE_PATH: '/'
CSRF_COOKIE_SAMESITE: 'Lax'
CSRF_COOKIE_SECURE: False
CSRF_FAILURE_VIEW: 'django.views.csrf.csrf_failure'
CSRF_HEADER_NAME: 'HTTP_X_CSRFTOKEN'
CSRF_TRUSTED_ORIGINS: []
CSRF_USE_SESSIONS: False
DATABASES: {
    'default': {   'ATOMIC_REQUESTS': False,
                   'AUTOCOMMIT': True,
                   'CONN_MAX_AGE': 0,
                   'ENGINE': 'django.db.backends.sqlite3',
                   'HOST': '',
                   'NAME': PosixPath('/home/.../tests/pyauto/src/db.sqlite3'),
                   'OPTIONS': {},
                   'PASSWORD': '********',
                   'PORT': '',
                   'TEST': {   'CHARSET': None,
                               'COLLATION': None,
                               'MIGRATE': True,
                               'MIRROR': None,
                               'NAME': None},
                   'TIME_ZONE': None,
                   'USER': ''}}
DATABASE_ROUTERS: '********'
[...]
DEBUG: True
DEBUG_PROPAGATE_EXCEPTIONS: False
DECIMAL_SEPARATOR: '.'
DEFAULT_CHARSET: 'utf-8'
DEFAULT_EXCEPTION_REPORTER: 'django.views.debug.ExceptionReporter'
DEFAULT_EXCEPTION_REPORTER_FILTER: 'django.views.debug.SafeExceptionReporterFilter'
DEFAULT_FILE_STORAGE: 'django.core.files.storage.FileSystemStorage'
DEFAULT_FROM_EMAIL: 'webmaster@localhost'
DEFAULT_HASHING_ALGORITHM: 'sha256'
DEFAULT_INDEX_TABLESPACE: ''
DEFAULT_TABLESPACE: ''
DISALLOWED_USER_AGENTS: []
EMAIL_BACKEND: 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST: 'localhost'
[...]
EMAIL_SUBJECT_PREFIX: '[Django] '
EMAIL_TIMEOUT: None
EMAIL_USE_LOCALTIME: False
EMAIL_USE_SSL: False
EMAIL_USE_TLS: False
FILE_UPLOAD_DIRECTORY_PERMISSIONS: None
FILE_UPLOAD_HANDLERS: ['django.core.files.uploadhandler.MemoryFileUploadHandler',
 'django.core.files.uploadhandler.TemporaryFileUploadHandler']
FILE_UPLOAD_MAX_MEMORY_SIZE: 2621440
FILE_UPLOAD_PERMISSIONS: 420
FILE_UPLOAD_TEMP_DIR: None
FIRST_DAY_OF_WEEK: 0
FIXTURE_DIRS: []
FORCE_SCRIPT_NAME: None
FORMAT_MODULE_PATH: None
FORM_RENDERER: 'django.forms.renderers.DjangoTemplates'
IGNORABLE_404_URLS: []
INSTALLED_APPS: ['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'crispy_forms',
 'django_celery_results',
 'django_celery_beat',
 'users.apps.UsersConfig',
 'pages.apps.PagesConfig',
 'mailgoblin.apps.MailgoblinConfig']
INTERNAL_IPS: []
LANGUAGES: [...]
LANGUAGES_BIDI: ['he', 'ar', 'ar-dz', 'fa', 'ur']
LANGUAGE_CODE: 'en-us'
LANGUAGE_COOKIE_AGE: None
LANGUAGE_COOKIE_DOMAIN: None
LANGUAGE_COOKIE_HTTPONLY: False
LANGUAGE_COOKIE_NAME: 'django_language'
LANGUAGE_COOKIE_PATH: '/'
LANGUAGE_COOKIE_SAMESITE: None
LANGUAGE_COOKIE_SECURE: False
LOCALE_PATHS: []
LOGGING: {
 }
LOGGING_CONFIG: 'logging.config.dictConfig'
LOGIN_REDIRECT_URL: 'home'
LOGIN_URL: '/accounts/login/'
LOGOUT_REDIRECT_URL: 'home'
MANAGERS: []
MEDIA_ROOT: ''
MEDIA_URL: '/'
MESSAGE_STORAGE: 'django.contrib.messages.storage.fallback.FallbackStorage'
MIDDLEWARE: ['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']
MIGRATION_MODULES: {
 }
MONTH_DAY_FORMAT: 'F j'
NUMBER_GROUPING: 0
PASSWORD_HASHERS: '********'
PASSWORD_RESET_TIMEOUT: '********'
PASSWORD_RESET_TIMEOUT_DAYS: '********'
PREPEND_WWW: False
ROOT_URLCONF: 'pyauto.urls'
SECRET_KEY: '********'
SECURE_BROWSER_XSS_FILTER: False
SECURE_CONTENT_TYPE_NOSNIFF: True
SECURE_HSTS_INCLUDE_SUBDOMAINS: False
SECURE_HSTS_PRELOAD: False
SECURE_HSTS_SECONDS: 0
SECURE_PROXY_SSL_HEADER: None
SECURE_REDIRECT_EXEMPT: []
SECURE_REFERRER_POLICY: 'same-origin'
SECURE_SSL_HOST: None
SECURE_SSL_REDIRECT: False
SERVER_EMAIL: 'root@localhost'
SESSION_CACHE_ALIAS: 'default'
SESSION_COOKIE_AGE: 1209600
SESSION_COOKIE_DOMAIN: None
SESSION_COOKIE_HTTPONLY: True
SESSION_COOKIE_NAME: 'sessionid'
SESSION_COOKIE_PATH: '/'
SESSION_COOKIE_SAMESITE: 'Lax'
SESSION_COOKIE_SECURE: False
SESSION_ENGINE: 'django.contrib.sessions.backends.db'
SESSION_EXPIRE_AT_BROWSER_CLOSE: False
SESSION_FILE_PATH: None
SESSION_SAVE_EVERY_REQUEST: False
SESSION_SERIALIZER: 'django.contrib.sessions.serializers.JSONSerializer'
SETTINGS_MODULE: 'pyauto.settings'
SHORT_DATETIME_FORMAT: 'm/d/Y P'
SHORT_DATE_FORMAT: 'm/d/Y'
SIGNING_BACKEND: 'django.core.signing.TimestampSigner'
SILENCED_SYSTEM_CHECKS: []
STATICFILES_DIRS: ['/home/.../pyauto1/src/static']
STATICFILES_FINDERS: ['django.contrib.staticfiles.finders.FileSystemFinder',
 'django.contrib.staticfiles.finders.AppDirectoriesFinder']
STATICFILES_STORAGE: 'django.contrib.staticfiles.storage.StaticFilesStorage'
STATIC_ROOT: None
STATIC_URL: '/static/'
TEMPLATES: [{'APP_DIRS': True,
  'BACKEND': 'django.template.backends.django.DjangoTemplates',
  'DIRS': ['/home/.../pyauto1/src/templates'],
  'OPTIONS': {'context_processors': ['django.template.context_processors.debug',
                                     'django.template.context_processors.request',
                                     'django.contrib.auth.context_processors.auth',
                                     'django.contrib.messages.context_processors.messages']}}]
TEST_NON_SERIALIZED_APPS: []
TEST_RUNNER: 'django.test.runner.DiscoverRunner'
THOUSAND_SEPARATOR: ','
TIME_FORMAT: 'P'
TIME_INPUT_FORMATS: ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']
TIME_ZONE: 'America/New_York'
USE_I18N: True
USE_L10N: True
USE_THOUSAND_SEPARATOR: False
USE_TZ: True
USE_X_FORWARDED_HOST: False
USE_X_FORWARDED_PORT: False
WSGI_APPLICATION: 'pyauto.wsgi.application'
X_FRAME_OPTIONS: 'DENY'
YEAR_MONTH_FORMAT: 'F Y'
is_overridden: <bound method Settings.is_overridden of <Settings "pyauto.settings">>

Were you able to fix it? – ashwani kumar Oct 08 '21 at 08:34 — ashwani kumar, Oct 08 '21 at 08:34

Django + Celery + Flower seems to break RabbitMQ broker

Starting Celery without flower argument:

Starting Celery with flower argument:

To diagnose the problem I've tried:

0 Answers0

Starting Celery with `flower` argument: