Kubeflow pipeline fail to create container

Question

I'm running Kubeflow in a local machine that I deployed with multipass using these steps but when I tried running my pipeline, it got stuck with the message ContainerCreating. When I ran kubectl describe pod train-pipeline-msmwc-1648946763 -n kubeflow I found this on the Events part of the describe:

Events:
  Type     Reason       Age                    From                  Message
  ----     ------       ----                   ----                  -------
  Warning  FailedMount  7m12s (x51 over 120m)  kubelet, kubeflow-vm  Unable to mount volumes for pod "train-pipeline-msmwc-1648946763_kubeflow(45889c06-87cf-4467-8cfa-3673c7633518)": timeout expired waiting for volumes to attach or mount for pod "kubeflow"/"train-pipeline-msmwc-1648946763". list of unmounted volumes=[docker-sock]. list of unattached volumes=[podmetadata docker-sock mlpipeline-minio-artifact pipeline-runner-token-dkvps]
  Warning  FailedMount  2m22s (x67 over 122m)  kubelet, kubeflow-vm  MountVolume.SetUp failed for volume "docker-sock" : hostPath type check failed: /var/run/docker.sock is not a socket file

Looks to me like there is a problem with my deployment, but I'm new to Kubernetes and can't figure out what I supposed to do right now. Any idea on how to solve this? I don't know if it helps but I'm pulling the containers from a private docker registry and I've set up the secret according to this.

No expert in kubeflow pipelines, but this looks similar to what you getting: https://github.com/kubeflow/pipelines/issues/3407 — rock'n rolla, Apr 07 '21 at 15:26
Looks like it, thanks! I just checked and looks like I don't have docker installed, I'm installing right now and trying again — João Areias, Apr 07 '21 at 15:30

score 5 · Answer 1 · answered May 10 '21 at 19:39

5

You don't need to use docker. In fact the problem is with workflow-controller-configmap in kubeflow name space. You can edit it with

kubectl edit configmap workflow-controller-configmap -n kubeflow

and change containerRuntimeExecutor: docker to containerRuntimeExecutor: pns. Also you can change some of the steps and install kubeflow 1.3 in mutlitpass 1.21 rather than 1.15. Do not use kubelfow add-on (at least didn't work for me). You need kustomize 3.2 to create manifests as they mentioned in https://github.com/kubeflow/manifests#installation.

answered May 10 '21 at 19:39

Ali Akbari

51
3

This is the correct answer as of 02/2022. microk8s in snap, channel 1.21/stable + kustomize 3.2.0 + manifest installation of Kubeflow 1.4.1 with single command `while ! kustomize build example...` according to the `kubeflow/manifests`' README.md + this change did the trick for me. No GPU support yet though, which is a pity. At least not on Windows/WSL2. – Hendrik Wiese Feb 07 '22 at 18:49

score 2 · Accepted Answer · answered Apr 07 '21 at 16:20

2

There was one step missing which is not mentioned in the tutorial, which is, I have to install docker. I've installed docker, rebooted the machine, and now everything works fine.

answered Apr 07 '21 at 16:20

João Areias

1,192
11
41

Kubeflow pipeline fail to create container

2 Answers2

Linked