4

In EKS I am trying to use SecretProviderClass to provide secrets as environment variables to containers. I can see the secret mounted inside the container but no combination of key/names is allowing me to use it as an environment variable. Insode the container I can cat /mnt/secrets-store/awscredentials And see the output:

{"accesskey":"ABCDE12345","secretkey":"a/long/redacted5tring"}

My SecretProviderClass is below

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
  namespace: default
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "path/to/service/awscredentials"
          objectType: secretsmanager
          objectAlias: awscredentials
  secretObjects:
  - secretName: awscredentials
    type: Opaque
    data: 
    - objectName: accesskeyalias
      key: accesskey
    - objectName: secretkeyalias
      key: secretkey

and my deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myservice
  labels:
    team: devops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myservice
  template:
    metadata:
      labels:
        app: myservice
    spec:
      serviceAccountName: myservice
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "aws-secrets"
      containers:
      - name: myservice
        image: someimage:2
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
        env:
        - name: AWS_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: awscredentials
              key: accesskey

When I run the deployment without reference to the SecretKeyRef the container runs and I can see the secret under /mnt/secrets-store/awscredentials. However, trying to set the environment variable results in the pod stuck in Pending state and the message: Error: secret "awscredentials" not found I reckon I have mixed up the name and keys somewhere but I've spent hours trying every combination I can think of. What am I missing?

jonny
  • 508
  • 5
  • 11
  • `awscredentials` exist in same namespace, where you are deploying ? – Harsh Manvar Aug 01 '22 at 11:19
  • I am using namespace 'default' but when I run `kubectl get secrets -n default` I only see 3 service-account-tokens. No sign of `awscredentials` – jonny Aug 01 '22 at 11:24
  • that's the issue thn, secret is not created there and due to that deployment failing – Harsh Manvar Aug 01 '22 at 11:25
  • Thank you Harsh, I thought that the SecretProviderClass created the secret. Do I need another `kind` to create the secret from the SecretProviderClass? – jonny Aug 01 '22 at 11:35
  • Are ASCP secrets sync with kubernetes? Following links has details to sync the secrets https://aws.amazon.com/blogs/security/how-to-use-aws-secrets-configuration-provider-with-kubernetes-secrets-store-csi-driver/, https://www.eksworkshop.com/beginner/194_secrets_manager/sync_native_secrets_env/ once thats done you can refer secrets are env variables – Nataraj Medayhal Aug 01 '22 at 11:58

2 Answers2

6

I eventually got this sorted. I had followed the AWS documentation for installing the driver which included using a helm chart i.e.

helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver

However, the documentation failed to point out that with this helm chart you need to specifically set a value syncSecret.enabled=true - as in the code below.

helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true

I used helm to uninstall the secrets-store-csi-driver, then re-installed with syncSecret.enabled=true and immediately my secretsmanager secret was available via kubectl get secrets -n default. So if you can see the secrets inside the container in the mounted volume but you can't set them as environment variables you should check that you installed with this value so that the k8s secret is created. Otherwise the secretObjects > secretName section is not actioned.

jonny
  • 508
  • 5
  • 11
  • Are the YAML manifests in your question correct? I am trying to figure out how to mount all key-vals in an AWS secret as environment variables into a pod without having to list them all via `jmesPath` and your example is not working for me. In particular, the `accesskeyalias` and `secretkeyalias` object names are not defined anywhere..? – Martin Melka Feb 16 '23 at 14:03
  • does this help: ``` spec: provider: aws parameters: objects: | - objectName: "devops/google-oidc/google-oidc-secret" objectType: "secretsmanager" objectAlias: google-oidc-secret jmesPath: - path: clientID objectAlias: clientID - path: clientSecret objectAlias: clientSecret secretObjects: - secretName: google-oidc-secret type: Opaque data: - objectName: clientID key: clientID - objectName: clientSecret key: clientSecret ``` – jonny Feb 17 '23 at 15:49
  • I was limited in that ^ comment and it didn't format - hope you can decipher @MartinMelka – jonny Feb 17 '23 at 15:50
  • I did decipher that, thank you. It's what I suspected - the keys/objectNames have to be defined in both the `parameters.objects.jmesPath` _and_ `secretObjects.data` mappings. That's the repetition I was trying to avoid but I came to the conclusion it is simply not possible. For potential future readers - the workaround I went with was to not use the synced k8s secrets (`secretObjects`) and instead 1) mounted the JSON file into the pod and 2) customized the entrypoint script to first parse and source all the variables from the JSON file as bash environment variables. – Martin Melka Feb 21 '23 at 11:54
  • This worked for me, it should be documented. – Federico Peralta May 05 '23 at 17:09
  • @jonny Can you explain how you created serviceaccountname ? I am stuck on that ? – Deepak Tripathi May 31 '23 at 11:40
  • 1
    @DeepakTripathi To create his serviceAccountName, you have to create a simple Kubernetes ServiceAccount but more important with an annotation "eks.amazonaws.com/role-arn" to use an IAM Role you have to create yourself before. This IAM Role should be attached to a policy where you allow the Actions ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"] on resource ["arn:*:secretsmanager:*:*:secret:-??????"] – moueidat Sep 01 '23 at 11:53
0

Can you provide example how you got this working

instead 1) mounted the JSON file into the pod and 2) customized the entrypoint script to first parse and source all the variables from the JSON file as bash environment variables

@martin

Surya yaramada
  • 1
  • 1