Airflow - Cannot import custom plugin

Question

I followed the tutorial about plugins.

I looked online also and find : Can't import Airflow plugins

But the top answer doesn't help me either.

Here a simplifed version of my project :

airflow_home
├── dags
│   └── etl.py  
└── plugins
    ├── __init__.py
    └── operators
        ├── __init__.py
        └── dump_file.py

plugins/operators/dump_file.py

from airflow.models import BaseOperator
from airflow.utils.decorators import apply_defaults

class DumpCsvFileToPostgres(BaseOperator):
    [...]

# plugins/__init__.py

from airflow.plugins_manager import AirflowPlugin
from plugins.operators.dump_file import DumpCsvFileToPostgres

# Defining the plugin class
class CustomPlugin(AirflowPlugin):
    name = "custom_plugin"
    operators = [
        DumpCsvFileToPostgres
    ]
    helpers = []

dags/etl.py

# tried 
# from airflow.operators import DumpCsvFileToPostgres
# from airflow.operators.custom_plugin import DumpCsvFileToPostgres
from custom_plugin import DumpCsvFileToPostgres

[...]

I still get

Broken DAG: [/root/airflow/dags/etl_dag.py] No module named 'custom_plugin'

My webserver and scheduler are running with a docker-compose

version: '3.7'
services:
  [...]

  webserver:
    image: godatadriven/airflow:latest
    environment:
      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://${POSTGRES_USER:-airflow}:${POSTGRES_PASSWORD:-airflow}@postgres:5432/${POSTGRES_DB:-airflow}
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - WAIT_FOR=postgres:5432
    depends_on:
      - postgres
    volumes:
      - ./dags:/root/airflow/dags
      - ./plugins:/root/airflow/plugins
      - ./logs:/root/airflow/logs
      - ./environment.yml:/dependencies/environment.yml
    ports:
      - "8080:8080"
    command: upgradedb_webserver

  scheduler:
    image: godatadriven/airflow:latest
    environment:
      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://${POSTGRES_USER:-airflow}:${POSTGRES_PASSWORD:-airflow}@postgres:5432/${POSTGRES_DB:-airflow}
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - WAIT_FOR=webserver:8080
    depends_on:
      - webserver
    volumes:
      - ./dags:/root/airflow/dags
      - ./plugins:/root/airflow/plugins
      - ./logs:/root/airflow/logs
      - ./environment.yml:/dependencies/environment.yml
      - ./dataset/Iowa_Liquor_Sales.csv:/root/airflow/dataset/dataset.csv
    command: scheduler

While I've never used `Airflow` `plugin`s, from the [docs](https://airflow.apache.org/plugins.html#interface), I infer that the *import statement* should be `from airflow.operators.custom_plugin import DumpCsvToPostgres`. If you look closely, your current import statement takes no account of your operator being exported as an Airflow plugin. Its a regular python import statement, where the top-level module is not available directly in `PYTHONPATH`; hence the error — y2k-shubham, Aug 07 '19 at 03:52
That's why you can see those lines of code commented # tried # from airflow.operators import DumpCsvFileToPostgres# from airflow.operators.custom_plugin import DumpCsvFileToPostgres — Ragnar, Aug 07 '19 at 03:57
What is the value for `core__plugins_folder` in the airflow configuration? — nightgaunt, Aug 07 '19 at 06:07
In `dags/etl.py` , change `from custom_plugin import DumpCsvFileToPostgres` to `from plugins.operators.dump_file import DumpCsvFileToPostgres` — kvivek, Aug 09 '19 at 09:40
@nightgaunt the value is `plugins_folder = /root/airflow/plugins` and this is what I have pointed to my docker volumes `./plugins:/root/airflow/plugins` — Ragnar, Aug 11 '19 at 19:42

score 1 · Answer 1 · answered Aug 09 '19 at 09:30

1

The import statement in your etl.py should be from airflow.macros import custom_plugin

Also, check that the value of plugins_folder key in airflow.cfg should point to plugins directory.

answered Aug 09 '19 at 09:30

Nitin Pandey

649
1
9
27

score 1 · Answer 2 · answered Aug 09 '19 at 09:44

1

Out of many solutions, One possible solution can be:

In dags/etl.py , change this line

from custom_plugin import DumpCsvFileToPostgres

to

from plugins.operators.dump_file import DumpCsvFileToPostgres

answered Aug 09 '19 at 09:44

kvivek

3,321
1
15
17

Antoine Krajnc · Answer 3 · 2020-01-04T15:09:43.500

0

Hey have you seen this stack-overflow? It seems that there is a much simpler way to import custom operators

Can't import Airflow plugins

edited Jan 04 '20 at 15:09

answered Jan 03 '20 at 14:16

Antoine Krajnc

1,163
10
29

Airflow - Cannot import custom plugin

3 Answers3