0

I have created a 3 node azure kubernetes cluster using the following commands

az group create --name ResourceGroup --location canadacentral
az provider register -n Microsoft.ContainerService
az provider register -n Microsoft.Compute
az provider register -n Microsoft.Network

az aks create --resource-group ResourceGroup --name ReplicaSet --node-count 3 --kubernetes-version  1.8.7 --node-vm-size Standard_A0 --generate-ssh-keys

kubectl create -f https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-operator.yaml
kubectl create -f https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-storageclasses.yaml

Subsequently I have created a postgres stateful set as well which does not start since opensci is not installed on the kubelet.

Kubelet logs from Node-1 (where the pgset pod is scheduled)

I0313 05:42:41.910525    7845 reconciler.go:257] operationExecutor.MountVolume started for volume "pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3" (UniqueName: "kubernetes.io/iscsi/10.0.20.229:3260:iqn.2016-09.com.openebs.jiva:pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3:0") pod "pgset-0" (UID: "a9826973-2674-11e8-a384-0a58ac1f03e3") 
I0313 05:42:41.910605    7845 operation_generator.go:416] MountVolume.WaitForAttach entering for volume "pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3" (UniqueName: "kubernetes.io/iscsi/10.0.20.229:3260:iqn.2016-09.com.openebs.jiva:pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3:0") pod "pgset-0" (UID: "a9826973-2674-11e8-a384-0a58ac1f03e3") DevicePath ""
E0313 05:42:41.910744    7845 iscsi_util.go:207] iscsi: could not read iface default error: 
E0313 05:42:41.910815    7845 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/iscsi/10.0.20.229:3260:iqn.2016-09.com.openebs.jiva:pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3:0\"" failed. No retries permitted until 2018-03-13 05:44:43.910784094 +0000 UTC (durationBeforeRetry 2m2s). Error: MountVolume.WaitForAttach failed for volume "pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3" (UniqueName: "kubernetes.io/iscsi/10.0.20.229:3260:iqn.2016-09.com.openebs.jiva:pvc-a980c1e4-2674-11e8-a384-0a58ac1f03e3:0") pod "pgset-0" (UID: "a9826973-2674-11e8-a384-0a58ac1f03e3") : executable file not found in $PATH
E0313 05:43:12.080406    7845 kubelet.go:1628] Unable to mount volumes for pod "pgset-0_default(a9826973-2674-11e8-a384-0a58ac1f03e3)": timeout expired waiting for volumes to attach/mount for pod "default"/"pgset-0". list of unattached/unmounted volumes=[pgdata]; skipping pod
E0313 05:43:12.081262    7845 pod_workers.go:182] Error syncing pod a9826973-2674-11e8-a384-0a58ac1f03e3 ("pgset-0_default(a9826973-2674-11e8-a384-0a58ac1f03e3)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"pgset-0". list of unattached/unmounted volumes=[pgdata]

My question is whether there is a way to configure and ensure that the kubelet comes up by default with the openiscsi initiator utils installed and running

Jay
  • 51
  • 1
  • 6

1 Answers1

0

The following steps were followed to manually install iscsi-initiator in kubelet:

  • SSH into the Kubernetes Nodes
  • Identify the docker container running the kubelet using sudo docker ps.
  • Enter the kubelet container shell sudo docker exec -it kubelet_container_id bash
  • Install open-iscsi. apt-get update apt install -y open-iscsi
Kiran Mova
  • 113
  • 6
  • 1
    Yes thank you very much Kiran. This is what I did for a work around but the problem would reoccur if I did a scale up or perhaps the node restarted. It would be desirable if the azure Kubernetes kubelet container provided open-iscsi by default or atleast a means to chose if it needs to be running. – Jay Mar 24 '18 at 05:57