I am using Azure CSI secrets provider. I am deploying it correctly I feel. I am running a job that runs software I made with golang that does the following in this order.
- Creates a new Azure user managed identity (az go sdk)
- Gives the new user managed identity read access to secrets in my AZ vault (az go sdk)
- Creates a new Namespace using the go-client (client-go sdk)
- Creates a new service account in the namespace (client-go sdk)
- Creates a role binding to said service account (client-go sdk)
- Creates a federation between the managed identity and the service account (az go sdk)
- Creates a CSI secrets provider for Azure Vault (client-go sdk)
- Creates a deployment that pulls a container (another one I made) from ACR and uses the CSI driver to mount secrets from Azure Vault (client-go sdk)
Everything works as I want it to, but - I get a few of these errors when my deployment is kicked off..
Warning FailedMount 12m kubelet MountVolume.SetUp failed for volume "my-mount-name" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod my-new-namespace/my-new-pod-from-my-deployment, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to get keyvault client: failed to get authorizer for keyvault client: failed to acquire token: FromAssertion(): http call(https://login.microsoftonline.com/<my-az-tenant-id>/oauth2/v2.0/token)(POST) error: reply status code was 400:
{"error":"invalid_request","error_description":"AADSTS70021: No matching federated identity record found for presented assertion. Assertion Issuer: '<my-AKS-cluster-oidc-issue-url>'. Assertion Subject: 'system:serviceaccount:my-new-namespace:my-new-service-account'. Assertion Audience: 'api://AzureADTokenExchange'. https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation\r\nTrace ID: bac9ec79-1d93-491d-9098-e03d4ce08700\r\nCorrelation ID: 1db5358b-c445-4e60-93d3-48a07ef2bce3\r\nTimestamp: 2023-03-10 22:50:42Z","error_codes":[70021],"timestamp":"2023-03-10 22:50:42Z","trace_id":"bac9ec79-1d93-491d-9098-e03d4ce08700","correlation_id":"1db5358b-c445-4e60-93d3-48a07ef2bce3","error_uri":"https://login.microsoftonline.com/error?code=70021"}
basically, indicating that it cannot find a federation for between the service account I created and the managed identity I am using in the CSI definition. it throws the error from zero to numerous times while starting the pod. It almost always resolves itself after several tries.
I do a check in my logic to make sure that azure resources are deployed fully before moving forward (the managed identity, the federation, etc.) so I am certain that Azure has completed setting up the federation.
Why am I getting this error? It has something to do with Azure and not Kubernetes for sure, but I am not sure how I can set up my logic to mitigate this.
Here is my CSI provider yaml definiton.
kind: SecretProviderClass
metadata:
creationTimestamp: "2023-03-10T22:50:38Z"
generation: 1
name: my-csi-name
namespace: first-main-auth-fe
resourceVersion: "920653"
uid: c2af1ab3-6c13-4f04-8786-f883f6b8204b
spec:
parameters:
clientID: <the client id of the managed identity>
cloudName: ""
keyvaultName: my-vault-name
objects: |
array:
- |
objectName: aSecretName
objectType: secret
resourceGroup: my-vault-rg
subscriptionID: <my-subscription-id>
tenantId: <my-tenant-id>
usePodIdentity: "false"
useVMManagedIdentity: "false"
provider: azure
secretObjects:
- data:
- key: tls.key
objectName: vaultSecretNameForTls
- key: tls.crt
objectName: vaultSecretNameForTls
secretName: mySecretName
type: kubernetes.io/tls
status: {}
