0

The example code in the MWAA docs for connecting MWAA to EKS has the following:

#use a kube_config stored in s3 dags folder for now
kube_config_path = '/usr/local/airflow/dags/kube_config.yaml'

This doesn't make me think that putting the kube_config.yaml file in the dags/ directory is a sensible long-term solution.

But I can't find any mention in the docs about where would be a sensible place to store this file.

Can anyone link me to a reliable source on this? Or make a sensible suggestion?

LondonRob
  • 73,083
  • 37
  • 144
  • 201

1 Answers1

0

From KubernetesPodOperator Airflow documentation:

Users can specify a kubeconfig file using the config_file parameter, otherwise the operator will default to ~/.kube/config.

In a local environment, the kube_config.yaml file can be stored in specific directory reserved for Kubernetes (e.g. .kube, kubeconfig). Reference: KubernetesPodOperator (Airflow).

In the MWAA environment, where DAG files are stored in S3, the kube_config.yaml file can be stored anywhere in the root DAG folder (including any subdirectory in the root DAG folder, e.g. /dags/kube). The location of the file is less important than explicitly excluding it from DAG parsing via the .airflowignore file. Reference: .airflowignore (Airflow).

Example S3 directory layout:

s3://<bucket>/dags/dag_1.py
s3://<bucket>/dags/dag_2.py
s3://<bucket>/dags/kube/kube_config.yaml
s3://<bucket>/dags/operators/operator_1.py
s3://<bucket>/dags/operators/operator_2.py
s3://<bucket>/dags/.airflowignore

Example .airflowignore file:

kube/
operators/
Andrew Nguonly
  • 2,258
  • 1
  • 17
  • 23
  • Thanks Andrew. I'm not sure you need to add the config file to `.airflowignore` though, because it will only attempt to parse Python files for DAG definitions. – LondonRob Jun 15 '22 at 12:40
  • Ack, thanks for clarifying @LondonRob. In any case, this is how I've deployed kube config files in MWAA. – Andrew Nguonly Jun 15 '22 at 14:16