0

How to process SIGTERM in argo or kubeflow stage/node/component?

It's possible to catch SIGTERM if your python script launched with PID 1.

But in argo/kubeflow container PID 1 is occupied by 1 root 0:00 /var/run/argo/argoexec emissary -- bash -c set -eo pipefail; touch /tmp/9306d238a1214915a260b696e45390ad.step; sleep 1; echo "

p.s. Tried to use

container.set_lifecycle(V1Lifecycle(pre_stop=V1Handler(_exec=V1ExecAction([
                "pkill", "-15", "python"
            ]))))

But this setting doesn't leads to the correct SIGTERM forwarding. SIGTERM on process python3 appears immediately before the pod killing, after ~30 sec since the pod stop initialization.

ice1x
  • 21
  • 6

0 Answers0