MWAA Apache Airflow DAG error importing EcsOperator

Question

I am trying to deploy an Airfow DAG to MWAA.

My requirements.txt:

apache-airflow[amazon] == 3.2.0

I import EcsOperator like this:

from airflow.contrib.operators.ecs_operator import EcsOperator

However, I get this error:

Broken DAG: [/usr/local/airflow/dags/mydag.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/airflow/dags/mydag.py", line 4, in <module>
    from airflow.contrib.operators.ecs_operator import EcsOperator
ImportError: cannot import name 'EcsOperator' from 'airflow.contrib.operators.ecs_operator' (/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/ecs_operator.py)

What am I doing wrong here?

Answer 1

What am I doing wrong here?

You might be referencing a different version (1.10.12?) of the Airflow documentation.

airflow.contrib.operators.ecs_operator (1.10.12)

The documentation for 3.2.0 is here. You can import the EcsOperator like this:

from airflow.providers.amazon.aws.operators.ecs import EcsOperator

airflow.providers.amazon.aws.operators.ecs (3.2.0)

Answer 2

The correct requirements.txt:

(empty)

And the correct import:

from airflow.providers.amazon.aws.operators.ecs import ECSOperator

Note the casing!

Answer 3

There are several issues here so I'll compile a detailed answer since privious answers didn't cover all of them.

First, the updated import path (provider release 3.2.0) is:

from airflow.providers.amazon.aws.operators.ecs import EcsOperator

The reason this doesn't work for you is because you install the provider with extras as:

apache-airflow[amazon]

as explained in the provider extra docs when installing provider in that manner you get the provider version which was released at the time of the Airflow version that you are using. Thus you are not guaranteed to get the updated provider version. So in case you are using Airflow 2.2.4 (latest at the time of writing this answer) you will get Amazon provider version 3.0.0 which is not the most recent one.

To get updated provider you should install it as:

pip install apache-airflow-providers-amazon

if you like to pick a specific version then:

pip install apache-airflow-providers-amazon==3.2.0

Please note that you should always install from constraint files provided by Airflow. Example:

pip install "apache-airflow-providers-amazon" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt"

Note that the provider is referring to constraints-main which constantly updated and not to constraints-2.2.4 or any other specific Airflow version. You can read more about it in the doc about Installation and upgrading of Airflow providers separately.