0

I'm creating DAG in apache managed airflow workflows environment. DAG basiclly uses EMR operator , it creates EMR cluster, Run spark job defined in DAG and deletes the EMR cluster. I have taken the code from airflow website. This is the link for the code-https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_modules/airflow/providers/amazon/aws/example_dags/example_emr_job_flow_manual_steps.html

Error Message:[EMR cluster is getting created, but it shows below, Is the error is related to airflow]

2021-08-25 05:00:04,520] {{logging_mixin.py:104}} INFO - [2021-08-25 05:00:04,520] {{local_task_job.py:188}} WARNING - State of this instance has been externally set to skipped. Terminating instance.
[2021-08-25 05:00:04,600] {{process_utils.py:100}} INFO - Sending Signals.SIGTERM to GPID 1897
[2021-08-25 05:00:04,621] {{taskinstance.py:1265}} ERROR - Received SIGTERM. Terminating subprocesses.
[2021-08-25 05:00:04,702] {{taskinstance.py:1482}} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/sensors/base.py", line 243, in execute
    time.sleep(self._get_next_poke_interval(started_at, run_duration, try_number))
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1267, in signal_handler
    raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
[2021-08-25 05:00:04,725] {{taskinstance.py:1532}} INFO - Marking task as FAILED. dag_id=emr_job_flow_manual_steps_dag, task_id=watch_step, execution_date=20210824T030000, start_date=20210825T030008, end_date=20210825T050004
[2021-08-25 05:00:04,793] {{process_utils.py:66}} INFO - Process psutil.Process(pid=1897, status='terminated', exitcode=1, started='03:00:07') (1897) terminated with exit code 1

My Airflow Environment class:mw1.small

dhanabalan Rangasamy
  • 319
  • 4
  • 18

1 Answers1

0

Looks like your DAG simply timed out after 2 hours:

start_date=20210825T030008, end_date=20210825T050004

Unfortunately MWAA integration with other AWS services is not well documented, but my guess would be that the MWAA environment execution role has no permissions to operate the EMR cluster.

Check the role on which the MWAA operates and assign the required permissions as explained here:

https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/emr.html

In order to run the 2 examples successfully, you need to create the IAM Service Roles (EMR_EC2_DefaultRole and EMR_DefaultRole) for Amazon EMR.

You can create these roles using the AWS CLI: aws emr create-default-roles

Jacek Sztandera
  • 904
  • 6
  • 5