0

We are having trouble giving a container within a pipeline uploaded to Kubeflow access to a private custom docker image stored in a google container registry. We are running kubeflow on top of a kubernetes cluster run on minikube. Can someone help us understand how to add the access token/service account to the Kubeflow deployment? We have read a couple of docs that achieve this on a custom Kubernetes deployment but not on a Kubeflow deployment.

The error we get when running the pipeline on Kubeflow is: This step is in Pending state with this message: ImagePullBackOff: Back-off pulling image

This is the pipeline code that calls the image. enter image description here

Thank you!!

Federico K
  • 31
  • 4
  • 1
    The [how to ask](https://stackoverflow.com/help/how-to-ask) page specifically says "do not use screenshots for code", independent of the fact that you have variables declared that we cannot see (`dsl` and `gcp`), so there is no way someone can _guess_ how to help you – mdaniel Dec 17 '19 at 06:10
  • Take a look at [k8s docs on adding private docker repository](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-in-the-cluster-that-holds-your-authorization-token) or [this blog article](https://ryaneschinger.com/blog/using-google-container-registry-gcr-with-minikube/) and let me know if you find it useful. – Matt Dec 17 '19 at 10:46

1 Answers1

2

This is issues can occur in some scenarios like:

  • Your kubeflow setup (Kubernetes cluster) and GCR are in different project

  • No GCR secret for the ml-pipeline service account which is responsible to run the pipeline. (you can see this kubectl --namespace=kubeflow get serviceaccount)

In your case, I think it is the second scenario. Though the following path will work on both scenarios.

  1. Create service_account.json with sufficient permission (GCR needs storage permission so give 'Storage admin') using the GCP console
Select “API & Services” > “Credentials”Select “Create credentials” > “Services Account Key” > “Create New Services Account”
  1. Add a Kubernetes Secret in Kubernetes Cluster to access GCR
kubectl create secret docker-registry $SECRETNAME \       
--docker-server=https://gcr.io \                          
--docker-username=_json_key \                             
--docker-email=user@example.com \                          
--docker-password="$(cat ./service_account.json.json)"
#username should be _json_key
  • Above method is for default service account. But patch this in Kufelow namespace
kubectl --namespace=kubeflow create secret docker-registry $SECRETNAME \  
--docker-server=https://gcr.io \                          
--docker-username=_json_key \                             
--docker-email=user@example.com \                          
--docker-password="$(cat ./service_account.json.json)"
#username should be _json_key
  1. Patching GCR secret with respective service account
# For Kubeflow specific problem path pipeline-runner serviceaccount
kubectl --namespace=kubeflow patch serviceaccount pipeline-runner -p '{"imagePullSecrets": [{"name": "$SECRETNAME"}]}'
Akash Desarda
  • 704
  • 8
  • 6
  • I could avoid ImagePullBackOff but I got PodInitializing which does not end. https://stackoverflow.com/questions/74242663/kuebflow-pipelines-podinitializing-forever – Ryo Matsuzaka Oct 29 '22 at 03:42