0

I'm having trouble with health checks starting too early on a kubernetes pod with multiple containers. My pod is set up like this:

  • main-container (nodejs)
  • sidecar container (http proxy)

Currently the health checks are configured on the sidecar container, and end up hitting both containers (proxy, then main container).

If the main container starts quickly, then everything is fine. But if the sidecar starts quickly and the main container starts slowly (e.g. if the image needs to be pulled) then the initial liveness checks start on the sidecar before the other container has even started.

Is there a way of telling kubernetes: don't start running any probes (liveness or readiness checks) until all the containers in the pod have started?

I know I can use a startupProbe to be more generous waiting for startup: but ideally and to avoid other monitoring warnings, I'd prefer to suppress the health/liveness probes completely until all the containers have started.

Wytrzymały Wiktor
  • 11,492
  • 5
  • 29
  • 37
bsmedberg
  • 177
  • 2
  • 8
  • 2
    startupProbe does suppress the others until it passes. That's its whole job :) – coderanger Jun 10 '21 at 20:06
  • Yes, but to be clear: I was hoping to not have *any* probes run until all the containers start. – bsmedberg Jun 11 '21 at 21:04
  • How would it know if not for the startupProbe? – coderanger Jun 11 '21 at 21:34
  • Kubernetes knows the container state is still "Pending" and hasn't launched yet. – bsmedberg Jun 12 '21 at 01:20
  • It knows that because of the probes. Like once it calls into runC and the process starts _at all_ then it has no idea what "launched" means at that point. Containers in a pod are designed to be fairly independent, I think you're assuming a lot more knowledge and coordination than exists or will ever exist :) – coderanger Jun 12 '21 at 01:47

1 Answers1

0

Answering your question - yes, there is a way of doing so using startupProbe on your sidecar container pointing to your main application's opened port. As per the documentation all other probes (per container) are disabled if a startup probe is provided, until it succeeds. For more information about how to set up a startup probe visit here.

Jakub Siemaszko
  • 668
  • 3
  • 8
  • My main application doesn't bind to the outside world, it only binds to localhost. I thought a kube probe connected from "outside" so it wouldn't be able to check against that port? Although I'd really love to not bother running any probes until all the containers have started... what's the point anyway? – bsmedberg Jun 11 '21 at 21:06
  • OK, I understand but I'm afraid that using probes might be the only way to solve your issue considering your application's architecture and its dependencies. – Jakub Siemaszko Jun 14 '21 at 10:55