3

I am trying to connect to Firestore from code running on GKE Container. Simple REST GET api is working fine, but when I access the Firestore from read/write, I am getting Missing or insufficient permissions.

An unhandled exception was thrown by the application.
Info
2021-06-06 21:21:20.283 EDT
      Grpc.Core.RpcException: Status(StatusCode="PermissionDenied", Detail="Missing or insufficient permissions.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1623028880.278990566","description":"Error received from peer ipv4:172.217.193.95:443","file":"/var/local/git/grpc/src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Missing or insufficient permissions.","grpc_status":7}")
         at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()

Update I am trying to provide secret to pod with service account credentails. Here is the k8 file which deploys a pod to cluster with no issues when no secrets are provided and I can do Get Operations which don't hit Firestore, and they work fine.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: foo-worldmanagement-production
spec:
  replicas: 1
  selector:
    matchLabels:
       app: foo
       role: worldmanagement
       env: production
  template:
    metadata:
      name: worldmanagement
      labels:
        app: foo
        role: worldmanagement
        env: production
    spec:
      containers:
      - name: worldmanagement
        image: gcr.io/foodev/foo/master/worldmanagement.21
        resources:
          limits:
            memory: "500Mi"
            cpu: "300m"
        imagePullworld: Always
        readinessProbe:
          httpGet:
            path: /api/worldManagement/policies
            port: 80
        ports:
        - name: worldmgmt
          containerPort: 80

Now, if I try to mount secret, the pod never gets created fully, and it eventually fails

kind: Deployment
apiVersion: apps/v1
metadata:
  name: foo-worldmanagement-production
spec:
  replicas: 1
  selector:
    matchLabels:
       app: foo
       role: worldmanagement
       env: production
  template:
    metadata:
      name: worldmanagement
      labels:
        app: foo
        role: worldmanagement
        env: production
    spec:
      volumes:
      - name: google-cloud-key
        secret:
          secretName: firestore-key   
      containers:
      - name: worldmanagement
        image: gcr.io/foodev/foo/master/worldmanagement.21
        volumeMounts:
        - name: google-cloud-key
          mountPath: /var/
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/key.json        
        resources:
          limits:
            memory: "500Mi"
            cpu: "300m"
        imagePullworld: Always
        readinessProbe:
          httpGet:
            path: /api/worldManagement/earth
            port: 80
        ports:
        - name: worldmgmt
          containerPort: 80

I tried to deploy the sample application and it works fine.

If I keep only the following the yaml file, the container gets deployed properly

- name: google-cloud-key
            secret:
              secretName: firestore-key 

But once I add the following to yaml, it fails

    volumeMounts:
    - name: google-cloud-key
      mountPath: /var/
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /var/key.json      

And I can see in GCP events that the container is not able to find the google-cloud-key. Any idea how to troubleshoot this issue, i.e why I am not able to mount the secrets, I can bash into the pod if needed.

I am using multi stage docker file made of

From mcr.microsoft.com/dotnet/sdk:5.0 AS build 
FROM mcr.microsoft.com/dotnet/aspnet:5.0 AS runtime

Thanks

skjagini
  • 3,142
  • 5
  • 34
  • 63
  • Can you give a few more details on your setup? Looks like you are NOT using Workload Identity? And are you passing `GOOGLE_APPLICATION_CREDENTIALS` into your container/app? If you are running FIrestore in Datastore mode, then `roles/datastore.user` should work – Gari Singh Jun 08 '21 at 08:56
  • Hi @GariSingh, I have updated the question and I am trying to use secret keys, without much luck. Can you check now. – skjagini Jun 09 '21 at 05:41
  • Is there a `/var/key.json` file in the container, that has the correct credentials? – yoda_droid Jun 12 '21 at 12:07

1 Answers1

1

Looks like they key itself might not be correctly visible to the pod. I would start by getting into the pod with kubectl exec --stdin --tty <podname> -- /bin/bash and ensuring that the /var/key.json (per your config) is accessible and has the correct credentials.

The following would be a good way to mount the secret:

volumeMounts:
- name: google-cloud-key
  mountPath: /var/run/secret/cloud.google.com
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
  value: /var/run/secret/cloud.google.com/key.json

The above assumes your secret was created with a command like:

kubectl --namespace <namespace> create secret generic firestore-key --from-file key.json

Also it is important to check your Workload Identity setup. The Workload Identity | Kubernetes Engine Documentation has a good section on this.

yoda_droid
  • 361
  • 1
  • 7
  • Bam, it worked. My pods were created under production namespace, while I was creating secret keys without specifying any name namespace. If you create secret without namespace it still gets created, but its just not available for container, unfortunately its very hard to troubleshoot why the pod is not able to access the secret, even the GCP Editor (VS Code) shows that the secret is correct whey you try to edit the .yaml, it doesn't have knowledge that there is namespace conflict. – skjagini Jun 13 '21 at 16:11