Airflow is configured in my ubuntu server and its webserver and scheduler are configured to run as system daemons. I am recently facing an issue where Airflow webserver is failing to start and is getting killed minutes after restart with the following log:
Mar 29 12:59:56 ip-myubuntu airflow[7243]: [2022-03-29 12:59:56,538] {scheduler_job.py:953} INFO - Figuring out tasks to run in Pool(name=default_pool) w
Mar 29 12:59:56 ip-myubuntu airflow[7243]: [2022-03-29 12:59:56,540] {scheduler_job.py:981} INFO - DAG Bandwidth-Anomaly-Detection has 0/4 running
Mar 29 12:59:56 ip-myubuntu airflow[7243]: [2022-03-29 12:59:56,579] {scheduler_job.py:1031} INFO - Setting the following tasks to queued state:
Mar 29 12:59:56 ip-myubuntu airflow[7243]:
Mar 29 13:00:17 ip-myubuntu airflow[7243]: Process Process-10:
Mar 29 13:00:22 ip-myubuntu airflow[7243]: Traceback (most recent call last):
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
Mar 29 13:00:22 ip-myubuntu airflow[7243]: self.run()
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
Mar 29 13:00:22 ip-myubuntu airflow[7243]: self._target(*self._args, **self._kwargs)
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/local/lib/python3.5/dist-packages/airflow/utils/dag_processing.py", line 613, in _run_processor_m
Mar 29 13:00:22 ip-myubuntu airflow[7243]: processor_manager.start()
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/local/lib/python3.5/dist-packages/airflow/utils/dag_processing.py", line 844, in start
Mar 29 13:00:22 ip-myubuntu airflow[7243]: simple_dags = self.heartbeat()
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/local/lib/python3.5/dist-packages/airflow/utils/dag_processing.py", line 1208, in heartbeat
Mar 29 13:00:22 ip-myubuntu airflow[7243]: processor.start()
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/local/lib/python3.5/dist-packages/airflow/jobs/scheduler_job.py", line 183, in start
Mar 29 13:00:22 ip-myubuntu airflow[7243]: self._process.start()
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/lib/python3.5/multiprocessing/process.py", line 105, in start
Mar 29 13:00:22 ip-myubuntu airflow[7243]: self._popen = self._Popen(self)
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/lib/python3.5/multiprocessing/context.py", line 212, in _Popen
Mar 29 13:00:22 ip-myubuntu airflow[7243]: return _default_context.get_context().Process._Popen(process_obj)
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/lib/python3.5/multiprocessing/context.py", line 267, in _Popen
Mar 29 13:00:22 ip-myubuntu airflow[7243]: return Popen(process_obj)
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 20, in __init__
Mar 29 13:00:22 ip-myubuntu airflow[7243]: self._launch(process_obj)
Mar 29 13:00:22 ip-myubuntu airflow[7243]: File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 67, in _launch
Mar 29 13:00:22 ip-myubuntu airflow[7243]: self.pid = os.fork()
Mar 29 13:00:22 ip-myubuntu airflow[7243]: OSError: [Errno 12] Cannot allocate memory
Mar 29 13:00:23 ip-myubuntu airflow[7243]: [2022-03-29 13:00:23,491] {dag_processing.py:652} WARNING - DagFileProcessorManager (PID=7395) exited with exi
Mar 29 13:00:23 ip-myubuntu airflow[7243]: [2022-03-29 13:00:23,590] {scheduler_job.py:1352} ERROR - Exception when executing execute_helper
Mar 29 13:00:23 ip-myubuntu airflow[7243]: Traceback (most recent call last):
Mar 29 13:00:23 ip-myubuntu airflow[7243]: File "/usr/local/lib/python3.5/dist-packages/airflow/jobs/scheduler_job.py", line 1350, in _execute
Mar 29 13:00:23 ip-myubuntu airflow[7243]: self._execute_helper()
Mar 29 13:00:23 ip-myubuntu airflow[7243]: File "/usr/local/lib/python3.5/dist-packages/airflow/jobs/scheduler_job.py", line 1402, in _execute_helper
Mar 29 13:00:23 ip-myubuntu airflow[7243]: simple_dags = self.processor_agent.harvest_simple_dags()
Mar 29 13:00:23 ip-myubuntu airflow[7243]: File "/usr/local/lib/python3.5/dist-packages/airflow/utils/dag_processing.py", line 632, in harvest_simple_d
Mar 29 13:00:23 ip-myubuntu airflow[7243]: self._heartbeat_manager()
I tried restarting the webserver and scheduler multiple times. Also reduced the dag parallelism and concurrency values, still facing this issue. Any help is appreciated.
Thanks in advance.
