4

I have a directory structure as such:

airflow_dags
├── dags
│   └── hk  
│       └── hk_dag.py  
├── plugins
│   └── cse   
│       └── operators.py   
│           └── cse_to_bq.py   
└── test
   └── dags   
       └── dag_test.py

In the GCS bucket created by Cloud Composer, there's a plugin folder where I upload the cse folder.

Now in my hk_dag.py file if I import the plugin like this:

from plugins.cse.operators.cse_to_bq import CSEToBQOperator

and run my unit test, it passes, but in cloud composer I get a ModuleNotFoundError: No module named 'plugins' error message.

If I import the plugin like this in my hk_dag.py:

from cse.operators.cse_to_bq import CSEToBQOperator

My unit test fails with ModuleNotFoundError: No module named 'cse' but it works fine in Cloud Composer.

How do I resolve it?

juwa92
  • 43
  • 4

1 Answers1

5

In Airflow 2.0 to import your plugin you just need to do it directly from the operators module.

In your case, has to be something like:

from operators.cse_to_bq import CSEToBQOperator

But before that you have to change your folder structure to:

airflow_dags
├── dags
│   └── hk  
│       └── hk_dag.py  
├── plugins
│   └── operators   
│       └── cse   
│           └── cse_to_bq.py 
└── test
   └── dags   
       └── dag_test.py
Alvaro
  • 813
  • 8
  • 16
  • I'm hoping there is a typo in this and that it should say `from operators.cse.cse_to_bq import CSEToBQOperator`. (Notice the extra `.cse`) Can you please confirm this? Also, I don't see how I can get my local `composer-dev` environment *and* the remote environment to both work, since it now seems that they need different paths because the one living in Google Storage is "ignoring" the `plugins` folder. It's annoying that Airflow would have done this... – Mike Williamson Apr 03 '23 at 13:46