Terminate istio-proxy after cronjob completion

Question

I have a k8s cronjob run my docker image transaction-service.

It starts and gets its job done successfully. When it's over, I expect the pod to terminate but... istio-proxy still lingers there:

And that results in:

Nothing too crazy, but I'd like to fix it.

I know I should call curl -X POST http://localhost:15000/quitquitquit

But I don't know where and how. I need to call that quitquitquit URL only when transaction-service is in a completed state. I read about preStop lifecycle hook, but I think I need more of a postStop one. Any suggestions?

curl -X POST http://localhost:15000/quitquitquit you should call when your cron job finishes — Jurabek Azizkhujaev, Jun 18 '22 at 20:36
@JurabekAzizkhujaev I know! But how do I configure the deployment yaml in order to let that happen? How do I execute a "post-completion" command? — Fabio B., Jun 19 '22 at 21:19

score 4 · Answer 1 · answered Jun 20 '22 at 21:33

4

You have a few options here:

On your job/cronjob spec, add the following lines and your job immediately after:

command: ["/bin/bash", "-c"]
args:
 - |
   trap "curl --max-time 2 -s -f -XPOST http://127.0.0.1:15020/quitquitquit" EXIT
   while ! curl -s -f http://127.0.0.1:15020/healthz/ready; do sleep 1; done
   echo "Ready!"
   < your job >

Disable Istio injection at the Pod level in your Job/Cronjob definition:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  ...
spec:
  ...
  jobTemplate:
    spec:
      template:
        metadata:
          annotations:
            # disable istio on the pod due to this issue:
            # https://github.com/istio/istio/issues/11659
            sidecar.istio.io/inject: "false"

Note: The annotation should be on the Pod's template, not on the Job's template.

answered Jun 20 '22 at 21:33

Gabriel Robledo Ahumada

1,542
2
14

I can't go for 2 because I need the istio-proxy sidecar. And about 1: my container runs my custom docker image, which already has an entrypoint (a Go app). Would command+args be run after my app exits then? – Fabio B. Jun 21 '22 at 15:39
No, the entrypoint would be ignored, you need either to transform your entrypoint into Kubernetes command/args, or include the code provided into your Go Entrypoint. – Gabriel Robledo Ahumada Jun 22 '22 at 16:16
1

You could use a preStop handler for that: https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/ – Chris Jun 24 '22 at 10:46
If `set -e` is defined, the `trap` needs either `set +e` before curl or `|| true` after curl. Otherwise a failed curl command will cause the container exit with an error code and the Job will be retried. The curl command may fail if the istio sidecar is not available for /quitquitquit - even though the actual application container would have completed successfully. In other words, `trap "curl -s -f ... /quitquitquit || true" EXIT` – Petrus Repo Feb 08 '23 at 13:19

score 1 · Accepted Answer · answered Jun 27 '22 at 15:12

In my Dockerfile I put

ADD ./entrypoint.sh /entrypoint.sh
RUN ["chmod", "+x", "/entrypoint.sh"]
RUN apk --no-cache add curl
ENTRYPOINT ["/entrypoint.sh"]

My entrypoint.sh looks like this:

#!/bin/sh
/app/myapp && curl -X POST http://localhost:15000/quitquitquit

It works.

score 0 · Answer 3 · answered Jun 16 '22 at 13:47

0

You can use TTL mechanism for finished Jobs mentioned in kubernetes doc which help removing the whole pod.

answered Jun 16 '22 at 13:47

Nataraj Medayhal

980
1
2
12

2

I don't know in advance how much time my cronjob (scheduled to run every hour) will take to complete. Sometimes it's 5 minutes, sometimes it's 30. I can't set a fixed ttl. And my problem is about the fact only one of the pods (my container) does complete, while the other one remains active (istio-proxy) – Fabio B. Jun 16 '22 at 14:24

Terminate istio-proxy after cronjob completion

3 Answers3

Linked