0

we are currently in the process of migration to AWS MWAA, and I would like to enable possibility for feature development in our dags to multiple developers in one environment at the same time. The idea is to create 'workspaces' for new features under the dags directory, I would like to keep this process as simple as possible and as automated as possible.

For example in nonprod and production envionments we would have:

dags
|-common
|-dag1
|-dag2
|....

And in dev:

dags
|-common
|-feature_1
   |-dag1
   |-dag2
|-feature_2
   |-dag1
   |-dag2
|....

It is doable however the import statements has to be altered: from dag1.params import param1 -> feature1.dag1.params import param1 I would like to avoid this manual action and automating this would take a lot of overhead.

How can I provide some separation like that for devs in airflow? Are there any good practices around development lifecycle?

An alternative solution is to provide multiple small environments for devs however this is costly and adds more maintenance. Any advises?

Mateusz Pierzchała
  • 21
  • 5

1 Answers1

1

For local development, I use aws-mwaa-local-runner. It is a docker-compose environment that replicates MWAA.

We forked this repo and store our DAGs in it. We run Github Actions to deploy the DAGs/plugins/requirements.txt to S3 and we can easily use traditional feature branching like in any other repo.

https://github.com/aws/aws-mwaa-local-runner

holly.evans
  • 129
  • 4