0

Updating my CSI Secrets Store Driver as it is outlined here:

https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver

I had it previously working by usign secretObjects which I accomplished with:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aks-akv-secret-provider
spec:
  provider: azure
  secretObjects:
  - secretName: myapp-prod-secrets
    type: Opaque
    data:
    - objectName: SENDGRID-API-KEY
      key: SENDGRID_API_KEY
  parameters:
    usePodIdentity: "true"                                        
    keyvaultName: myappakvprod
    cloudName: ""                               
    objects: |
      array:
        - |
            objectName: SENDGRID-API-KEY             
            objectType: secret                 
            objectVersion: ""  
    tenantId: $tenantId

In the manifest for the API for example I would:

...
  env:
    - name: SENDGRID_API_KEY
      valueFrom:
        secretKeyRef:
          name: myapp-prod-secrets
          key: SENDGRID_API_KEY
  volumeMounts:
    - name: secrets-store01-inline
      mountPath: /mnt/secrets-store
      readOnly: true
volumes:
  - name: secrets-store01-inline
    csi: 
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: aks-akv-secret-provider
...

And then in the code I would just do something like and it would work:

EMAIL_KEY = os.environ['SENDGRID_API_KEY']

Basically, it is mounting the AKV secrets to the Pod and then creating Kubernetes secret environmental variables.

However, this method in the documentation seems like an after thought ("You might sometimes..." as opposed to "you must"):

https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver#sync-mounted-content-with-a-kubernetes-secret

I interpret this as indicating that simply mounting the secrets to the Pod is sufficient and that creating Kubernetes secrets of them is extraneous.

My question is this:

If you are just mounting the secrets to the Pod, how should you access them from the application code if they aren't environmental variables?

They can be accessed with:

kubectl exec busybox-secrets-store-inline -- cat /mnt/secrets-store/ExampleSecret

So I'd imagine in Python, JavaScript, etc. you could do what ever the analogous command is to read in the secret instead of os.environ['ExampleSecret']. But if you aren't mounting secrets the same way in local development, there's disparity in the code base.

Just looking for clarification on this.

cjones
  • 8,384
  • 17
  • 81
  • 175

1 Answers1

2

If you just mount the secrets to the pod, the secrets will be available as files on the mount location

  • The filename will be the name of the secret (or an alias you specify in the secret provider class)
  • The content of the file will be the secret value.

To access and use the secrets in your code you would need to read the files to retrieve the secret values. Many frameworks have built in functionality for mapping such structures into configuration objects, for example in .NET there is the Key-per-file configuration provider

But if you aren't mounting secrets the same way in local development, there's disparity in the code base.

Yes that is correct, and would be true for anything you do different in development and production. If you want to avoid disparities, use the same mechanism for providing secret values to the application in both development and production, either through environment variables or through files.

danielorn
  • 5,254
  • 1
  • 18
  • 31
  • Hmmmm.... so I'd need to figure out how to deploy CSI for local development to sync keys from AKV or at least make them available at `/mnt/secrets-store/` in the running container. Or just continue to utilize `secretKeyRef` and access them as environmental variables which standardizes it in the code base regardless of how the env vars are generated in the environment. Are there security issues with the latter? – cjones Mar 18 '22 at 13:25
  • Not sure I follow. When you do local development right now, how do you do it? Is it a secretKeyRef from a K8s secret? Does that secret get generated by CSI from a keyvault? – danielorn Mar 18 '22 at 13:35
  • Currently, for both production and development, I'm using `secretKeyRef` in the manifests. The difference is in production I'm using CSI and AKV/AKS integration. It syncs with AKV and creates a Kubernetes secret `myapp-prod-secrets` (first block of code I posted above). Development does not sync with AKV. I haven't started trying to figure that out yet. There is a script that prompts the dev for secrets and creates a `kubectl create secret ...` That way similar manifests can be used and the code base just looks for `os.environ[...]` to the env var from kubectl secrets. – cjones Mar 18 '22 at 14:45
  • Ah, I see. Couldn't the same apply if you are using mounts? I.e in production you mount using CSI which fetches from KeyVault, in local development you just mount the Kubernetes secret and project the keys to the paths that your application expect? https://kubernetes.io/docs/concepts/configuration/secret/#projection-of-secret-keys-to-specific-paths Or in local development you mount a local folder into the pod containing the keys. – danielorn Mar 18 '22 at 15:12
  • It seems that you are mostly concerned with testing the application locally (not the deployment or surrounding infrastructure?) in that case the primary objective should be to expose the keys to the application in the same way. For local development it might even be advisable to not involve Azure KeyvVault and CSI at all to avoid unnecessary external dependencies that require network connections etc – danielorn Mar 18 '22 at 15:14
  • 1
    Yeah I definitely feel like CSI for local development is probably unnecessarily complicated. Additionally it does feel like the additional step I'm currently doing of creating a Kubernetes secret for use as an env var is also unnecessary. I'll look into options and methods for mounting the secrets for local development. – cjones Mar 19 '22 at 13:52