4

We have a service that needs to run a few longer SQL queries when it shuts down. However, when the pod receives a SIGTERM from Kubernetes, the istio proxy container waits only 5s prior to shutting down. This causes our queries to fail and the service terminates ungracefully.

Things we've tried:

  • Setting the terminationGracePeriodSeconds to 3600. Istio still shuts down after 5s.
  • Keeping an HTTP connection open to try to force Istio not to shut down. Istio still shuts down, forcing our HTTP connection to close too.
  • Setting TERMINATION_DRAIN_DURATION_SECONDS to 3600 on the istio container. Istio keeps running until 3600s have elapsed, even if our service has finished shutting down. We tried calling curl -XPOST http://127.0.0.1:15000/quitquitquit to get Istio to shut down sooner but it remains running for the full time.

How can we get Istio to stay running long enough for our service to terminate gracefully, without having it stay running for too long?

Jesse Bye
  • 123
  • 2
  • 8

1 Answers1

4

As far as I know unfortunately Graceful shutdown for istio sidecar is not supported:

Currently there is not. This seems like an interesting feature request though, it may be worth a feature request on github.

I also think that setting up an appropriate thread on github will be a solution. The only way I know is to use TERMINATION_DRAIN_DURATION_SECONDS option.

There are also several topic on github related to this theme:

istio-proxy stop before my containers #10112

And currently envoy does not support graceful shutdown. xref: envoyproxy/envoy#2920 Once envoy implements this, we can support it in sidecar.

Render istio-proxy environment variables from global.proxy.env or environmentOverride annotation #18333

For anyone looking through issues for why their connection to a pod is disconnecting early, hopefully this helps.

You need the pod to have a terminationGracePeriodSeconds, which I assume you were already aware about, and you also need an annotation for the pod's sidecar config:

annotations: # https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#ProxyConfig proxy.istio.io/config: | terminationDrainDuration: {{ $terminationGracePeriodSeconds }}s

Envoy shutting down before the thing it's wrapping can cause failed requests #7136

Graceful shutdown with injected envoy-sidecar #536

Wytrzymały Wiktor
  • 11,492
  • 5
  • 29
  • 37
Mikołaj Głodziak
  • 4,775
  • 7
  • 28
  • 1
    Yes, I believe so. We're going to try adding a `preStop` command to the istio-proxy container and see if that lets us keep it running long enough for our service to terminate gracefully. – Jesse Bye Sep 13 '21 at 19:50
  • @WytrzymałyWiktor and @MikołajGłodziak , doesn't `terminationGracePeriodSeconds` solve the problem already as you mentioned in the answer? – user2734550 Oct 17 '21 at 05:56
  • @JesseBye were you able to add the preStop to the sidecar? – Saeid Ghafouri Apr 28 '23 at 00:32