AWS MWAA (Managed Apache Airflow) where to put the python code used in the dags?

Question

Where do you put your actual code? The dags must be thin, this assumes that when the task starts to run it would do the imports, and run some python code.

When we were on the standalone airflow I could add to the PYTHON_PATH my project root and do the imports from there, but in the AWS managed airflow I don't find any clues.

score 0 · Accepted Answer · answered Sep 11 '21 at 14:53

0

Put your DAGs into S3. Upon initialization of your MWAA environment, you will determine the S3 bucket containing your code.

E.g., create a bucket <my-dag-bucket> and place your DAGs in a subfolder dags

s3://<my-dag-bucket>/dags/

Also make sure to define all python dependencies in a requirements file and put that one in the same bucket as well:

s3://<my-dag-bucket>/requirements.txt

Finally, if you need to provide own modules, zip them up and put the zip file in the bucket, too:

s3://<my-dag-bucket>/plugins.zip

See https://docs.aws.amazon.com/mwaa/latest/userguide/get-started.html

answered Sep 11 '21 at 14:53

dovregubben

364
2
16

And what you do when you have multiple projects which you want to run at airflow? I can't zip all of those in a plugins.zip – Ivailo Bardarov Oct 21 '21 at 07:33
Do you have multiple python modules? Why can't you put them into the plugins.zip ? – dovregubben Oct 22 '21 at 13:08
I have multiple teams which work on different projects. Each team has its own dags and dependencies. – Ivailo Bardarov Oct 24 '21 at 08:38
Have you tried to put all your code in the S3 dags folder? Not sure if this works but I suppose MWAA will sync any files placed there – dovregubben Oct 24 '21 at 16:03
2

There was one .airflowignore file where I can ignore everything which is not DAG. Thank you! – Ivailo Bardarov Oct 25 '21 at 11:57

AWS MWAA (Managed Apache Airflow) where to put the python code used in the dags?

1 Answers1