0

kubernetes and kubeflow were all set on my GPU service, while the errors occurred when I tried to start up a tutorials pipeline, it looks like stuck in something about pvc and pv.

here are the details:

...

    Environment Variables from:
      metadata-grpc-configmap  ConfigMap  Optional: true
    Environment:
      WORKFLOW_ID:     (v1:metadata.labels['workflows.argoproj.io/workflow'])
      KFP_POD_NAME:   parameterized-tfx-oss-s7krr-2814409879 (v1:metadata.name)
      KFP_NAMESPACE:  kubeflow-user-example-com (v1:metadata.namespace)
      WORKFLOW_ID:     (v1:metadata.labels['workflows.argoproj.io/workflow'])
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-editor-token-qn279 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  podmetadata:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  docker-sock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  Socket
  mlpipeline-minio-artifact:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  mlpipeline-minio-artifact
    Optional:    false
  default-editor-token-qn279:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-editor-token-qn279
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                    From            Message
  ----     ------       ----                   ----            -------
  Warning  FailedMount  43m (x65 over 13h)     kubelet, admin  Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[mlpipeline-minio-artifact default-editor-token-qn279 podmetadata docker-sock]: timed out waiting for the condition
  Warning  FailedMount  34m (x73 over 14h)     kubelet, admin  Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[default-editor-token-qn279 podmetadata docker-sock mlpipeline-minio-artifact]: timed out waiting for the condition
  Warning  FailedMount  30m (x58 over 14h)     kubelet, admin  Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[docker-sock mlpipeline-minio-artifact default-editor-token-qn279 podmetadata]: timed out waiting for the condition
  Warning  FailedMount  15m (x421 over 14h)    kubelet, admin  MountVolume.SetUp failed for volume "mlpipeline-minio-artifact" : secret "mlpipeline-minio-artifact" not found
  Warning  FailedMount  5m19s (x175 over 14h)  kubelet, admin  Unable to attach or mount volumes: unmounted volumes=[mlpipeline-minio-artifact], unattached volumes=[podmetadata docker-sock mlpipeline-minio-artifact default-editor-token-qn279]: timed out waiting for the condition
秋水泡茶
  • 7
  • 1

1 Answers1

-1

Here I found the approach of this question. The k8s1.19 was using to deploy kubeflow1.3, while the problem gone when I falling back to k8s1.17. That is it.

秋水泡茶
  • 7
  • 1