How to process SIGTERM in argo or kubeflow stage/node/component?
It's possible to catch SIGTERM if your python script launched with PID 1.
But in argo/kubeflow container PID 1 is occupied by
1 root 0:00 /var/run/argo/argoexec emissary -- bash -c set -eo pipefail; touch /tmp/9306d238a1214915a260b696e45390ad.step; sleep 1; echo "
p.s. Tried to use
container.set_lifecycle(V1Lifecycle(pre_stop=V1Handler(_exec=V1ExecAction([
"pkill", "-15", "python"
]))))
But this setting doesn't leads to the correct SIGTERM forwarding. SIGTERM on process python3 appears immediately before the pod killing, after ~30 sec since the pod stop initialization.