0

I am starting out with Google Cloud Composer environment. I want to organize my projects as follow:

dags/
├── project-1
|   ├── dist
|   |   ├── my_custom-0.0.1-py3-none-any.whl
|   ├── src
|   |   ├── project_1_dag.py
|   ├── config
|   |   ├── config.ini
|   ├── test
|   |   ├── test_dag.py
├── project-2
|   ├── dist
|   |   ├── my_custom-0.0.2-py3-none-any.whl
|   ├── src
|   |   ├── project_2_dag.py
|   ├── config
|   |   ├── config.ini
|   ├── test
|   |   ├── test_dag.py
└── project-n

The whl file contains the common utility classes and functions that I wrote for services like BigQuery, GCS, paramiko etc.

By following the above structure, I can isolate the dependencies and their versions, apply changes and improvements to existing classes and functions over time without affecting other projects.

As per the airflow documentation, I can not use PythonVirtualOperator since most of the my codebase are classes. I did not find any other reference to achieve this.

My questions are follow:

  • How can I achieve this?
  • How to install and manage different versions of whl files in same composer environment?

Please note:

  • My company's security policy does not allow me to host these whl files on public PyPI repo.
  • I have already looked into importing from local dependencies approach, but I would like to explore whl versioning approach first for the reason I mentioned above.
kshaikh
  • 41
  • 1
  • 8
  • Have you tried anything yet? Can you further explain what you mean with "manage different versions of whl files"? – Ricco D Jun 07 '22 at 02:42
  • Thanks @RiccoD. We want to avoid the frequent production code changes where the existing utility class functions change require us to update all of the existing project DAGs where they are being used. We design our classes as generic as possible, but sometimes it required us to go back and change the base class. Hence we are managing multiple versions of whl files to create some sort of isolation. This also help us to avoid any potential operational issues if we somehow miss to update the one of the existing DAG. – kshaikh Jun 07 '22 at 15:13
  • @RiccoD We already tried this [approach](https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html#typical-structure-of-packages) – kshaikh Jun 07 '22 at 15:29

0 Answers0