1

I have defined readiness and liveness probes for the container in a Kubernetes deployment. When these fail I expect to see their output included in an event for the pod. However I don't see such events. I do see other events for the same pod.

I am running on GKE, with Kubernetes v1.6.4.

The code that should produce the events is here: https://github.com/kubernetes/kubernetes/blob/v1.6.4/pkg/kubelet/prober/prober.go#L94.

What I see instead is logs indicating that a "ref" for the container could not be retrieved. I see similar logs for other places in the code that try to record container-related events.

Is this a bug? These events are potentially very useful. In particular it's the only way to see output from a failing probe so that I can diagnose probes that are failing unexpectedly.

[I'm not certain if this is an appropriate issue for StackOverflow. Please let me know if it would be better raised as an issue in Github.]

Ben Butler-Cole
  • 2,011
  • 1
  • 17
  • 23
  • I also have a GKE 1.6.4 cluster. I followed [this example](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-a-liveness-command) from the Kubernetes docs, and after about 40 seconds, I did see an even when the pod was killed for failing its liveness probe. It looked like: ```"liveness-exec_kube-system(4339b96a-5b7e-11e7-956e-42010a8a0fdc)" container "liveness" is unhealthy, it will be killed and re-created. ``` – Eric Tune Jun 27 '17 at 21:25
  • If you are able to, you could SSH to the node that the pod is running on, and check if the `!hasRef` condition is being hit. You'd grep the kubelet logs for "No ref for container". – Eric Tune Jun 27 '17 at 21:29
  • @EricTune Thank you. I have checked that it is indeed the "no ref for container" problem. – Ben Butler-Cole Jun 29 '17 at 09:57
  • Filed an issue: https://github.com/kubernetes/kubernetes/issues/49063 – Eric Tune Jul 17 '17 at 23:09

1 Answers1

0

This was a bug that has been fixed in Kubernetes 1.6.7.

Ben Butler-Cole
  • 2,011
  • 1
  • 17
  • 23