0

After successful deployment of CAM (Was up and running for couple of days), suddenly "cam-mongo" microservice went down and while checking the logs for pod using below 2 command it gives you Error synching pod

1) kubectl describe pods -n services

Warning BackOff 3s (x3 over 18s) kubelet, 9.109.191.126 Back-off restarting failed container

Warning FailedSync 3s (x3 over 18s) kubelet, 9.109.191.126 Error syncing pod

With this information you don't know what went wrong and how do you fix it

2) kubectl -n services logs cam-mongo-5c89fcccbd-r2hv4 -p (with -p option you can grab the logs from previously running container ) The above command show below information:

exception in initAndListen: 98 Unable to lock file: /data/db/mongod.lock Resource temporarily unavailable. Is a mongod instance already running?, terminatingConclusion:

While starting the container inside "cam-mongo" pod it was unable to use the existing /data/db/mongod.lock file and hence your pod will be not up and running and you cannot access CAM

Gian Filippo Maniscalco
  • 33
  • 8

1 Answers1

0

After further analysis I resolved the issue as following:

1) spin up a container and mount the cam-mongo volume within it.

To do this I used the below pod creation yaml which will mount the concern pv's where /data/db/ is present.

kind: Pod

apiVersion: v1

metadata:

name: mongo-troubleshoot-pod

spec:

volumes:

  • name: cam-mongo-pv

    persistentVolumeClaim:

    claimName: cam-mongo-pv

containers:

  • name: mongo-troubleshoot

    image: nginx

    ports:

    • containerPort: 80

      name: "http-server"

    volumeMounts:

    • mountPath: "/data/db"

      name: cam-mongo-pv

RUN:kubectl -n services create -f ./mongo-troubleshoot-pod.yaml

2) Use "docker exec -it /bin/bash " (look for it from "kubectl -n services describe po/mongo-troubleshoot-pod-xxxxx" info)

cd /data/db

rm mongod.lock

rm WiredTiger.lock

3) kill the pod which you have created for troubleshooting

4) kill the corrupted cam-mongo pod using below command

kubectl delete pods -n services

It fixed the issue.

Gian Filippo Maniscalco
  • 33
  • 8