1

I have two things which are not immediately compatible:

  1. The old stuff: Several Jenkins Build-server jobs that must be performed, once or several times daily, currently configured with the cron to trigger at specific times. These jobs sometimes run for many hours, and their runtime is not predictable.
  2. The new stuff: A global IT department who insists that they must be able to patch and reboot any jenkins server and/or agent as they wish - although within some known maintenance windows.
    • The window can be described in cron syntax as "30 15 19-25 * 6" to "00 20 19-25 * 6"

The addition of the maintenance breaks things in at least these ways:

  • Jenkins may not be running when a job is supposed to start.
  • Jenkins jobs that are running are terminated unexpectedly.

In both cases whats needed is a method to detect if a given job was started at a specific time and allowed to "naturally" complete its course. In either case the job should be scheduled as soon as possible thereafter.

I've naturally asked the IT-Department to allow us to do this in a way that's compatible to our systems, but that's not going to happen in this decade. :-(

I'm now considering how to deal with it. My thoughts currently are:

  1. To minimise damage
    • Use Windows cron to shut down the Jenkins service well before the maintenance window - ensuring it can not start during the maintenance window, and to start up the Jenkins service following the maintenance period. (I'm here thinking to trigger two scripts, which will -indirectly- modify the service startup setting in the registry.)
  2. To fix damage, have a separate Jenkins job run on first boot following maintenance, performing the following using a groovy script
    • Scan a set of known jobs for jobs that ended prematurely at the time the service window started.
    • Scan a set of known jobs for jobs that ought to have started during the maintenance window,
  3. For each job found in step 2, rebuild those one by one, prioritized based on their original start time.

But this looks like a complicated task, and potentially a hell to administer.

What other options do I have?

Flimzy
  • 2,454
  • 18
  • 26
fsteff
  • 161
  • 8
  • This sounds exactly like the use case for anacron. – jayhendren Jan 25 '18 at 17:01
  • Thank you, @jayhendren - this is indeed a job for something like anacron. I just need to find out how to integrate it in Jenkins. – fsteff Jan 25 '18 at 21:30
  • 1
    Well you could trigger a jenkins job out of an anacron job. Or you could write your job so that it runs hourly, but only runs to completion if it hasn't yet done so in the past 24 hours. Or you could write a Jenkins plugin that basically re-implements anacron. And so on. – jayhendren Jan 25 '18 at 23:10

0 Answers0