0

I have a Python script, part of a test system that calls many third party tools/processes on multiple [Windows] machines, and hence has been designed to clean up comprehensively/carefully when aborted with CTRL-C; the clean-up can take many seconds, depending on what's going on. This clean-up process works fine from a [Windows] command prompt.

I run that Python script from [a scripted pipeline] Jenkinsfile, using return_value = bat("python my_script.py params", returnStatus: true), which also works fine.

However I need to be able to perform the abort/clean-up during a Jenkins [v2.263.4] run, i.e. when someone presses the little red X, and that bit I can't fathom. I understand that Jenkins sends SIGTERM when the abort button is pressed so I am trapping that in my_script.py:

SAVED_SIGTERM_HANDLER = signal(SIGTERM, sigterm_handler)

...and running the processes I would normally call from a KeyboardInterrupt in sigterm_handler() as well, but they aren't being called. I understand that the IO stream to the Jenkins console stops the moment the abort button is pressed; I can see that the clean-up functions aren't being called by looking at the behaviour of my script(s) from the "other side": it appears as though my_script.py is simply stopping dead, all connections from it drop the moment the abort button is pressed, there is no clean-up.

Can anyone suggest a way of making the abort button in Jenkins give my bat()ed Python script time to clean-up? Or am I just doing something wrong? Or is there some other approach to this within Jenkins that I'm missing?

Rob
  • 865
  • 1
  • 8
  • 21

2 Answers2

1

You should be able to use a "post" action to execute any clean up needed: https://www.jenkins.io/doc/book/pipeline/syntax/#post

I know that doesn't take into account the cleanup logic you already have but it's probably the safest thing to do. Maybe separate out the cleanup logic into a separate script and make it idempotent and then you can call it no matter what at the end of a pipeline and if it has already run then it should do nothing if run again.

Max Schenkelberg
  • 805
  • 4
  • 5
  • Unfortunately that will be too late: one of the third party executables that the script calls is "touchy"; the support team for it tell me that the only way to stop it safely is to send it "exit\n" over stdin. So I do need to do the clean-up "in-line", if you like, there and then. – Rob Jun 21 '21 at 23:11
0

After much figuring out, and kudos to our tools people who found the critical "cookie" implementation detail in Jenkins, the workaround to take control of the abort process [on Windows] is as follows:

  • have Jenkins call a wrapper, let's call it (a), and open a socket or a named-pipe (socket would work on both Linux and Windows),
  • (a) then launches (b), via "start" so that (b) runs as a separate process but, CRITICALLY, the environment that (a) passes to (b) MUST have JENKINS_SERVER_COOKIE="ignore" added to it; Jenkins uses this flag to find the processes it has launched in order to kill them, so you must set this "cookie" to "ignore" to stop Jenkins killing (b),
  • (b) connects back to (a) via the socket or pipe,
  • (a) remains running for as long as (b) is connected to the socket or pipe but also lets itself be killed by CTRL-C/SIGTERM,
  • (b) then launches the thing you actually want to run,
  • when (a) is terminated by a Jenkins abort (b) notices (because the socket or pipe will close) and performs a controlled shut-down of the thing you wanted to run before (b) exits,
  • separately, make a thing, let's call it (c), which checks whether the socket/named-pipe is present: if it is then (b) hasn't terminated yet,
  • in Jenkinsfile, wrap the calling of (a) in a try()/catch()/finally() and call (c) from the finally(), hence ensuring that the Jenkins pipeline only finishes when (b) has terminated (you might want to add a guard timer for safety).

Quite a thing, and all for the lack of what would be a relatively simple API in Jenkins.

Rob
  • 865
  • 1
  • 8
  • 21