this is my first post asking for help with airflow. In my company we were using version 1.10.15 of Airflow with some custom modules saved in a customs folder inside the DAGs folder and everything worked fine, I'm upgrading to version 2.1.0, and because of that I want to follow good practices and turn that custom folder into custom plugins which Airflow treats by default. So far so good, what happens is that I can't find a clear documentation that works for the version I'm using (2.1.0), I always find documentations of past versions or that don't even mention the version but I couldn't make it work, I'll try put below how my environment is configured as well as an example of what I need.
My environment is set up like this:
dags
└── my_dag.py
plugins
├── __init__.py
├── hooks
│ ├── __init__.py
│ └── my_hook.py
It is configured like this because I followed the instructions on this link and the astronomer guide. Before they ask the plugins_folder configuration in the airflow.cfg file is pointing to the correct folder.
Now let's have my questions:
- Is it necessary to create folders inside the plugins folder separating hooks, operators and sensors?
- is it necessary to have a
__init__.pyfile in every folder inside plugins? I put the files in blank like the link I gave above, but without success, if that's what the content of the files would be? could you give me examples? - Is it necessary to execute the
airflow pluginsCLI command? I executed the command and the return is
airflow@ip:/opt/airflow$ airflow plugins
/usr/local/lib/python3.6/dist-packages/airflow/providers/http/operators/http.py:26 DeprecationWarning: This decorator is deprecated.
In previous versions, all subclasses of BaseOperator must use apply_default decorator for the`default_args` feature to work properly.
In current version, it is optional. The decorator is applied automatically using the metaclass.
No plugins loaded
- is there any other way to do this natively and correctly?
Well to sum up I'm really lost, I apologize because besides not being fluent in English I'm studying python for just over 3 months, so basic things may be going unnoticed by me, I would be very grateful if someone could give me a light with some code examples, if you need more details on how I'm doing the codes I'm ready to give you all the details, thanks and I'm waiting
[update]
After a lot of trying and researching, I finally figured out how to get plug-ins recognized, followed the Amazon Managed Workflows for Apache Airflow guide and everything workeds well. In short, you need to follow the folder structure with an init.py file in each folder and the central point is the my_airflow_plugin.py file which points to each custom plugin. Well, at least it was registered for those who have the same doubts in the future.
