0

I have airflow configured with celery worker and the celery broker is redis. All the services are up and i am monitoring through flower for the celery workers and tasks.

when i run this query in the airflow database 

select count(*), executor_class, min(start_date), max(start_date) from airflow.job where
start_date > '2020-01-17' group by executor_class;

i see a lot of jobs having executor_class as NoneType :

enter image description here

I have configured airflow as required in the 1.10.6 documentation. But everyjob has stopped running on CeleryExecutor since 2020-01-17 05:38. This was the time airflow was upgraded and it assigns NoneType after the upgrade to 1.10.6 from 1.10.1. Also monitoring in the flower shows that all the celery workers are active but they are not sending any messages through the broker.

airflow config

celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_log_server_port = 8793
broker_url = redis://localhost:6379/0
result_backend = redis://localhost:6379/0
flower_host = 0.0.0.0
flower_port = 5555
default_queue = default
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
executor = CeleryExecutor
dag_concurrency = 16
task_runner = StandardTaskRunner

what should i do to make sure the jobs are assigned 'CeleryExecutor' so that the DAGs starts running (how to get celery brokers to assign these tasks to celery workers) ?

Tejus Prasad
  • 6,322
  • 7
  • 47
  • 75

1 Answers1

0

To fix this problem, the number of executors and parallelization factor has to be increased. Setting the parallelism to more than 250 fixed the issue as we have 1000s of tasks in 100s of active DAGs.

Tejus Prasad
  • 6,322
  • 7
  • 47
  • 75