1

I've been trying to deploy an app to google cloud (GKE) and am running into issues. The cluster is in the same project per Error: Status 403 trying to pull repository. kubernetes 1.5.4 specific and I've set up "Storage Object Viewer" permissions on all the service accounts listed in IAM permissions page. I'm at a loss as to what the issue might be or how to use the message to troubleshoot it.

Here is the output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.9-gke.1900", GitCommit:"008fd38bf3dc201bebdd4fe26edf9bf87478309a", GitTreeState:"clean", BuildDate:"2021-04-14T09:22:08Z", GoVersion:"go1.15.8b5", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.21) and server (1.19) exceeds the supported minor version skew of +/-1

Here is the events table from a kubectl describe <pod> command I ran:

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  24s               default-scheduler  Successfully assigned default/my-app-prod-6b5d68655f-9kc9h to gke-my-app-prod-default-pool-6a4bc86f-dfc9
  Normal   BackOff    19s               kubelet            Back-off pulling image "gcr.io/project-demo-123456/my_app:prod"
  Warning  Failed     19s               kubelet            Error: ImagePullBackOff
  Normal   Pulling    5s (x2 over 22s)  kubelet            Pulling image "gcr.io/project-demo-123456/my_app:prod"
  Warning  Failed     5s (x2 over 19s)  kubelet            Failed to pull image "gcr.io/project-demo-123456/my_app:prod": rpc erro
r: code = Unknown desc = failed to pull and unpack image "gcr.io/project-demo-123456/my_app:prod": failed to resolve reference "gc
r.io/project-demo-123456/my_app:prod": unexpected status code [manifests prod]: 403 Forbidden
  Warning  Failed     5s (x2 over 19s)  kubelet            Error: ErrImagePull
Nona
  • 5,302
  • 7
  • 41
  • 79
  • are you manually able to pull the image ? Eg: `docker pull .....` – P.... Jul 08 '21 at 18:05
  • Yes, I can after following the directions at https://cloud.google.com/container-registry/docs/advanced-authentication – Nona Jul 08 '21 at 18:47
  • It looks like, that you have a problem with permissions. Did you see [this site](https://cloud.google.com/kubernetes-engine/docs/troubleshooting#console)? – Mikołaj Głodziak Jul 09 '21 at 08:18
  • Did you have any luck solving this? I've run into the same issue (not fixed by any of the suggested docs) - 403 when pulling images from GCR. GKE cluster is the same project as GCR. I noticed that the cluster has had this issue since 13 days ago - same as the day you posted.. coincidence? or did something change on GKE? – localghost Jul 22 '21 at 15:25
  • you need to do `docker login`, get the generadted credentials file, and create a secret. Then you need to use that secret in your yaml to pull the image. – suren Jul 23 '21 at 15:40

1 Answers1

0

Not sure if the comments from others helped you resolve this already, but I had the same issue here. Checked the service account, recreated the cluster, re-enabled the Kubernetes API, nothing was working.

TL;DR:

Manually create a key from the service account and add into the cluster, per this guide, from the section starting 'Create & use GCR credentials'.

localghost
  • 329
  • 1
  • 4
  • 11