0

I am using kubernetes to start java pods. The pod startup delay vary between 10 seconds and about a minute depending on the load of the node, the time flyway took to migrate the tables, ...

To avoid having kubernetes killing the pods that are starting we set the liveness probe with an initial delay of two minutes.

It saves us from pods being eternally killed because they start too slowly. But in case of scaling up, crash recovery, we loose a couple of seconds / minutes before the freshly started pod join the service.

Is there any way to optimize that ? A way to tell kubernetes "we are live, you can start using the liveness probe" before the initial delay ?

poussma
  • 7,033
  • 3
  • 43
  • 68

2 Answers2

0

For starters, this will not happen at all. Liveness probe does not control how pods are joined to the service, as you stated, it will restart container if it fails to satisfy the probe, but it will not make the service await for successfull liveness probe before it is added as a service endpoint. For that you have a separate readiness probe. So this should be a no issue for you (btw. you might want to use both readiness and liveness probes to get optimal process)

Radek 'Goblin' Pieczonka
  • 21,554
  • 7
  • 52
  • 48
0

I think you need to reduce the work the container is doing. You mention the database migrations. It may be better to supply them as a one time job to Kubernetes and not trying to run them at every start. Effectively for a certain version of your software you only do them once and each subsequent start still has to do the work of checking if the database schema is already up to date.

Heiko Rupp
  • 30,426
  • 13
  • 82
  • 119