4

I am trying to pass secret variables to my KubernetesPodOperator in airflow

Here is what I have done :

  1. Create a secret.yaml file that looks like the following
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  SECRET_1: blabla 
  SECRET_2: blibli 
  1. Apply the secret :
kubectl apply -f ./secret.yaml
  1. Retrieve the secret from my DAG file :
from airflow.contrib.kubernetes.secret import Secret
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.models import DAG

SECRET_1 = Secret(
   deploy_type="env", deploy_target="SECRET_1", secret="ai-controller-object-storage", key="SECRET_1"
)
SECRET_2 = Secret(
   deploy_type="env", deploy_target="SECRET_2", secret="ai-controller-object-storage", key="SECRET_2"
)

with DAG(...) as dag:
   KubernetesPodOperator(
           task_id=..,
           trigger_rule="all_success",
           namespace="default",
           image=IMAGE,
           startup_timeout_seconds=600,
           secrets=[
               SECRET_1,
               SECRET_2], ...)

So now as I understand, I should access SECRET_1 as environnement variable in my container from KubernetesPodOperator

However my first task from a python script (with os.environ["SECRET_1"]) returns an error that indicates that this environment variable does not exist :

KeyError: 'SECRET_1'

How can I access this variable from my python script then ?

Luis Blanche
  • 557
  • 9
  • 18
  • 1
    Apparently this was fixed with airflow 2 – Luis Blanche May 19 '21 at 06:57
  • What is the pip package required to import `from airflow.contrib.kubernetes.secret import Secret`? I am getting a error `ModuleNotFoundError: No module named 'airflow.contrib.kubernetes'` – Kavya May 12 '22 at 07:18
  • you need to install airflow with the kubernetes extension : ```pip install airflow[kubernetes]``` – Luis Blanche May 12 '22 at 08:31
  • It is already installed, yet I'm unable to import the Secret – Kavya May 12 '22 at 10:12
  • 1
    Oh right, this post was made with airflow 1.xx, if you are using airflow 2.0 and above the import should be : ```from airflow.kubernetes.secret import Secret``` cf : https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_modules/airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.html – Luis Blanche May 12 '22 at 11:56
  • Has anyone come up with a solution for this in Airflow 1.xx? – Matt Jul 25 '22 at 19:16

0 Answers0