Make Airflow load all keys in a Kubernetes secret without specifying the keys

Question

I am using Google Cloud Composer 1.17.7 with Airflow 2.1.4. I am mainly following these docs.

I created a Kubernetes secret that looks like this:

apiVersion: v1
data:
  KEY1: base64encodedvalue1
  KEY2: base64encodedvalue2
  KEY3: base64encodedvalue3
  ...
  KEYN: base64encodedvalueN
kind: Secret
metadata:
  creationTimestamp: "2021-12-13T12:58:03Z"
  name: airflow-secrets
  namespace: default
  resourceVersion: "*****"
  uid: *****
type: Opaque

To load this secret and use it in Airflow as environment variables, it seems to me that I have to create an airflow.kubernetes.secret.Secret object for every key included in the K8s secret.

secret_env1 = Secret(
    deploy_type='env',
    deploy_target='KEY1',
    secret='airflow-secrets',
    key='KEY1')
secret_env2 = Secret(
    deploy_type='env',
    deploy_target='KEY2',
    secret='airflow-secrets',
    key='KEY2')
...
secret_envN = Secret(
    deploy_type='env',
    deploy_target='KEYN',
    secret='airflow-secrets',
    key='KEYN')

This becomes cumbersome when there are a lot of keys to be used. I came up with a for loop solution which still forces me to specify all keys needed:

secret_envvars = [
    'KEY1',
    'KEY2',
    'KEY3',
    ...
    'KEYN'
]
secret_envs = [
    Secret(
        deploy_type='env',
        secret='airflow-secrets',
        deploy_target=var,
        key=var,
    ) for var in secret_envvars]

I wonder if there is a way to load all keys of a K8s secret in Airflow without the need to explicitly specify the keys. For example, it would be helpful to be able to programmatically access the list of all keys included in the K8s secret (something like k8sSecret.keys().

not but could be helpful : https://github.com/apache/airflow/blob/main/airflow/kubernetes/secret.py#L79 — Harsh Manvar, Dec 15 '21 at 12:19
I don't see how that method can solve my problem, why not answer with a code snippet demonstrating how to use it? — Giulia Savorgnan, Dec 21 '21 at 08:04

score 0 · Answer 1 · edited Aug 28 '22 at 15:43

0

secret_all_keys  = Secret('env', None, 'airflow-secrets-2')

Reference: https://airflow.apache.org/docs/apache-airflow/1.10.12/kubernetes.html

edited Aug 28 '22 at 15:43

Tyler2P

2,324
26
22
31

answered Aug 23 '22 at 18:14

Евгений Лисевич

1

Make Airflow load all keys in a Kubernetes secret without specifying the keys

1 Answers1