KFServing pod "error: container storage-initializer is not valid"

Question

I am new to KFServing and Kubeflow.

I was following https://github.com/kubeflow/kfserving/tree/master/docs/samples/v1alpha2/tensorflow to deploy a simple inference service.

However, when looking at the logs, I am unable to find the container storage-initializer. The only containers my predict service pod has are kfserving and queue-proxy.

I am currently on Kubeflow 1.2 and Kubernetes 1.17 on IBM Cloud.

Error Message Image

score 0 · Answer 1 · answered Feb 27 '21 at 20:45

storage-initializer is an init container, so if you describe the pod you won't find it in the containers section of pod spec but in the initContainers section.

$ kubectl get pod flowers-sample-predictor-default-00002-deployment-58bb9557sf7g2 -o json | jq .status.initContainerStatuses
[
  {
    "containerID": "docker://e40e5f86401b3715118b873fec4ae6c3ef57765ffbb5c9ab48757234c4f53b6f",
    "image": "gcr.io/kfserving/storage-initializer:v0.5.0",
    "imageID": "docker-pullable://gcr.io/kfserving/storage-initializer@sha256:1d396c0c50892f5562a1c24d925691ec786e5d48e08200f3f9bb17bb48da40ae",
    "lastState": {},
    "name": "storage-initializer",
    "ready": true,
    "restartCount": 0,
    "state": {
      "terminated": {
        "containerID": "docker://e40e5f86401b3715118b873fec4ae6c3ef57765ffbb5c9ab48757234c4f53b6f",
        "exitCode": 0,
        "finishedAt": "2021-02-27T20:13:25Z",
        "reason": "Completed",
        "startedAt": "2021-02-27T20:13:11Z"
      }
    }
  }
]

I'm not familiar with the model label you are using, can you retry by using the app label or the pod name directly?

$ kubectl logs -l app=flowers-sample-predictor-default-00002 -c storage-initializer
[I 210227 20:13:12 initializer-entrypoint:13] Initializing, args: src_uri [gs://kfserving-samples/models/tensorflow/flowers] dest_path[ [/mnt/models]
[I 210227 20:13:12 storage:43] Copying contents of gs://kfserving-samples/models/tensorflow/flowers to local
[W 210227 20:13:15 _metadata:104] Compute Engine Metadata server unavailable onattempt 1 of 3. Reason: timed out
[W 210227 20:13:15 _metadata:104] Compute Engine Metadata server unavailable onattempt 2 of 3. Reason: [Errno 113] No route to host
[W 210227 20:13:18 _metadata:104] Compute Engine Metadata server unavailable onattempt 3 of 3. Reason: timed out
[W 210227 20:13:18 _default:250] Authentication failed using Compute Engine authentication due to unavailable metadata server.
[I 210227 20:13:19 storage:127] Downloading: /mnt/models/0001/saved_model.pb
[I 210227 20:13:19 storage:127] Downloading: /mnt/models/0001/variables/variables.data-00000-of-00001
[I 210227 20:13:25 storage:127] Downloading: /mnt/models/0001/variables/variables.index
[I 210227 20:13:25 storage:76] Successfully copied gs://kfserving-samples/models/tensorflow/flowers to /mnt/models

KFServing pod "error: container storage-initializer is not valid"

1 Answers1