1

I'm new to Celery, and I created a simple app that connects to a web socket server to receive tasks and schedule them using Celery. My Celery queue display tasks based on the type of the message (text messages first, and then buttons that run the next task if one of them is clicked). Locally, everything run as expected. But, some tasks are repeated multiple times in production, especially the triggers (buttons). In my production environment, I created a web service for Django and one Celery background worker with a Redis database.

Here are the commands I used to run Celery worker and beat in production:

# Start command (celery worker and beat)
celery -A bot.celery worker --beat --scheduler django --loglevel=info --concurrency 4
# Start command (Django)
daphne -b 0.0.0.0 bot.asgi:application

My Django and Celery settings:

CELERY_BROKER_URL = "redis://localhost:6379/0" # localhost replaced with the internal Redis URL in production
CELERY_RESULT_BACKEND = "redis://localhost:6379/1" # localhost replaced with the internal Redis URL in production

TIME_ZONE = 'UTC'

CELERY_ENABLE_UTC = True
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'
ASGI_APPLICATION = 'bot.asgi.application'

Some of the worker output:

Nov 1 09:00:15 AM  [2022-11-01 08:00:15,178: INFO/MainProcess] missed heartbeat from celery@srv-cdcl4202i3msb94icl70-5469f7b9d8-gzks7
Nov 1 09:05:33 AM  [2022-11-01 08:05:33,207: INFO/MainProcess] Task schedule_chat_message[574703ea-51d8-4acf-b9e9-202c37d0cbe5] received
Nov 1 09:05:33 AM  [2022-11-01 08:05:33,211: INFO/MainProcess] Task schedule_chat_message[ef2b2dd0-f297-4480-9577-da83a44bc244] received
Nov 1 09:05:33 AM  [2022-11-01 08:05:33,223: INFO/MainProcess] Task schedule_chat_message[a3619feb-9768-4e6b-b8be-966c335eae5f] received
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,182: INFO/ForkPoolWorker-2] Task schedule_chat_message[a3619feb-9768-4e6b-b8be-966c335eae5f] succeeded in 0.9560789109673351s: None
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,272: INFO/ForkPoolWorker-3] Task schedule_chat_message[574703ea-51d8-4acf-b9e9-202c37d0cbe5] succeeded in 1.0607709939940833s: None
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,280: INFO/ForkPoolWorker-4] Task schedule_chat_message[ef2b2dd0-f297-4480-9577-da83a44bc244] succeeded in 1.0533532970002852s: None
Nov 1 09:05:34 AM  [2022-11-01 08:05:34,936: INFO/Beat] DatabaseScheduler: Schedule changed.
Nov 1 09:05:35 AM  [2022-11-01 08:05:35,274: INFO/Beat] Scheduler: Sending due task Scheduled to send to chat_1ea56ac at 2022-11-01 08:05:33.211860+00:00 (send_out_chat_message)
Nov 1 09:05:35 AM  [2022-11-01 08:05:35,289: INFO/MainProcess] Task send_out_chat_message[37854a6c-d20a-45b1-99ba-37b22d818af4] received
Nov 1 09:05:35 AM  [2022-11-01 08:05:35,562: INFO/ForkPoolWorker-3] Task send_out_chat_message[37854a6c-d20a-45b1-99ba-37b22d818af4] succeeded in 0.27148923301137984s: None
Nov 1 09:05:38 AM  [2022-11-01 08:05:38,090: INFO/Beat] Scheduler: Sending due task Scheduled to send to chat_1ea56ac at 2022-11-01 08:05:33.227478+00:00 (send_out_chat_message)
Nov 1 09:05:38 AM  [2022-11-01 08:05:38,096: INFO/MainProcess] Task send_out_chat_message[80ca730b-0ff4-43df-b3ec-2758b0b493df] received
Nov 1 09:05:38 AM  [2022-11-01 08:05:38,276: INFO/ForkPoolWorker-3] Task send_out_chat_message[80ca730b-0ff4-43df-b3ec-2758b0b493df] succeeded in 0.1784440579940565s: None
Nov 1 09:05:44 AM  [2022-11-01 08:05:44,359: INFO/Beat] Scheduler: Sending due task Scheduled to send to chat_1ea56ac at 2022-11-01 08:05:33.226634+00:00 (send_out_chat_message)
Nov 1 09:05:44 AM  [2022-11-01 08:05:44,365: INFO/MainProcess] Task send_out_chat_message[a1bcff89-78af-41b6-8e81-95aa5d3172d3] received
Nov 1 09:05:44 AM  [2022-11-01 08:05:44,561: INFO/ForkPoolWorker-3] Task send_out_chat_message[a1bcff89-78af-41b6-8e81-95aa5d3172d3] succeeded in 0.19544574100291356s: None
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,089: INFO/MainProcess] Task schedule_chat_message[a83a6683-1750-422f-8494-7226152249b4] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,099: INFO/MainProcess] Task schedule_chat_message[d983a9d0-2243-4a87-9844-5f2e986f7626] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,103: INFO/MainProcess] Task schedule_chat_message[f30d05ec-257f-4516-bfcb-5880b85d3e85] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,109: INFO/MainProcess] Task schedule_chat_message[f8e1a7a3-21a9-41f3-a8d9-9ceae4f0c10f] received
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,864: INFO/ForkPoolWorker-3] Task schedule_chat_message[a83a6683-1750-422f-8494-7226152249b4] succeeded in 0.773912389006s: None
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,869: INFO/ForkPoolWorker-4] Task schedule_chat_message[d983a9d0-2243-4a87-9844-5f2e986f7626] succeeded in 0.7689848589943722s: None
Nov 1 09:05:50 AM  [2022-11-01 08:05:50,968: INFO/ForkPoolWorker-2] Task schedule_chat_message[f30d05ec-257f-4516-bfcb-5880b85d3e85] succeeded in 0.8630019359989092s: None

Your help is much appreciated!

Irikio
  • 23
  • 6

2 Answers2

1

Since you are using Redis as broker, it seems to be related to this celery issue. It looks like developers using SQS may encounter a similar problem.

txicos
  • 249
  • 3
  • 6
0

I was having same issue, we managed to solve the problem... enter image description here

the problem was that we schedule the task in django (m/h/dm/my/d) * 13 * * 1-5.

We made this to run the task at 13pm every day of week, but in django we need to specify minutes!

after editing the schedule to (m/h/dm/my/d) 00 13 * * 1-5, problem solved!

Thiago
  • 1