0

Im trying to work with custom operators in managed amazon airflow (MWAA). the tick is that my custom operator relaying on amazon provider package that installed via pip upon setup (requirements.txt)

The operator I'm inheritance from is EmrContainerOperator

Plugins folder within the zip I uploaded (In my case no needed hooks)

plugins
|-- __init__.py
`-- operators
    |-- __init__.py
    `-- my_operator.py

Within the init I defined the plugin by extending the AirflowPlugin

from airflow.plugins_manager import AirflowPlugin

from operators.my_operator import MyOperator


class MyPlugin(AirflowPlugin):
    name = 'my_plugin'
    operators = [MyOperator]

requirments.txt file content

apache-airflow-providers-snowflake==3.3.0
apache-airflow-providers-amazon==2.4.0
dag-factory==0.15.0
apache-airflow-providers-slack

But Im getting annoying error:

Broken plugin: [/usr/local/airflow/plugins/operators/mys_operator.py] No module named 'airflow.providers.amazon.aws.operators.emr'
Benny Elgazar
  • 243
  • 2
  • 9
  • 1
    After 2 days I finally understand what's the issue. mwaa uses 2.4.0 indeed but its not enough to just change it after you develop on the latest. the emr_container operator i used was moved to different file path and the class renamed. therefor i couldn't find it in the mwaa. – Benny Elgazar Oct 10 '22 at 07:27
  • Please post this as a separate answer and mark it as solved :) – Jorrick Sleijster Oct 10 '22 at 09:39

2 Answers2

0

After 2 days I finally realized what's the issue was. MWAA comes pre-packed with AWS provider version 2.4.0. The emr_container operator I used (on the latest provider I installed) was moved to different file path and the class renamed. therefor I couldn't find the class in the MWAA.

Benny Elgazar
  • 243
  • 2
  • 9
0

the new path for EMR operator in MWAA 2.4.3 is

from airflow.providers.amazon.aws.operators.emr import ( EmrAddStepsOperator, EmrCreateJobFlowOperator )

Anandkumar
  • 1,338
  • 13
  • 15