0

I am trying to use class EcsCreateClusterOperator in MWAA but I get the following error:

Broken DAG: [/usr/local/airflow/dags/etl_basic_ecs/etl_ecs_basic_dag.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/airflow/dags/etl_basic_ecs/etl_ecs_basic_dag.py", line 5, in <module>
    from airflow.providers.amazon.aws.operators.ecs import (
ImportError: cannot import name 'EcsCreateClusterOperator' from 'airflow.providers.amazon.aws.operators.ecs' (/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/ecs.py)

This library works perfectly in my local environment installed with:

pip install apache-airflow-providers-amazon==5.0.0

I also added this library (version 5.0.0) to my requirements.txt file before deploying to MWAA, in the UI seems like the provider is installed:

Screenshot of installed Amazon Provider 5.0.0

The log of the update shows that the library is downloaded:

Installing library log

Here is the documentation of the class: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/operators/ecs/index.html this class was introduced recently (version 5.0.0).

By default MWAA (Airflow 2.2.2) has the Amazon Provider with an older version (2.4), seems like Airflow is really loading the older version of the library.

Any help would be appreciated, thank you.

Armando Fa Menor
  • 1
  • Hi. Did you find a resolution to this issue? I am experiencing the same problem with attempting to import the LambdaHook class. – dataEng Sep 09 '22 at 07:56
  • Hello, I realized that MWAA installs the new version of the Amazon Provider library in a different path and It is only available inside Python or Bash operators (tested with "pip freeze" and "pip show"), unfortunately when I import the library directly in the DAG then MWAA imports the default version. – Armando Fa Menor Sep 10 '22 at 08:29
  • Thanks for the reply. I was able to reference an older class (AwsLambdaHook) within version 2.4.0 – dataEng Sep 12 '22 at 12:42

2 Answers2

0

MWAA with Airflow version 2.2.2 is constrained to apache-airflow-providers-amazon==2.4.0. See the constraints file for more details.

References:

  1. Airflow Versions (MWAA)
  2. Constraints file
Andrew Nguonly
  • 2,258
  • 1
  • 17
  • 23
  • Not working. even if you try to use the 2.4.0 version – Benny Elgazar Oct 10 '22 at 06:37
  • @BennyElgazar, Note: the `EcsCreateClusterOperator` operator does not exist in `apache-airflow-providers-amazon==2.4.0`. `2.4.0` operator reference: https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.4.0/operators/index.html – Andrew Nguonly Oct 10 '22 at 17:35
0

I had the same issue with EksCreateClusterOperator:

from airflow.providers.amazon.aws.operators.eks import ( ImportError: cannot import name 'EksCreateClusterOperator' from 'airflow.providers.amazon.aws.operators.eks' (/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/eks.py)

Looks like is case sensitive. So instead of using EcsCreateClusterOperator change it to ECSCreateClusterOperator (capital ECS) how it is showing on airflow github. That fixed my issue.

Azhar Khan
  • 3,829
  • 11
  • 26
  • 32
Manuel Israel De Aguinaga
  • 1