Halt the puppet master while Exec is running a script

Question

I am fairly new to puppet and thinking if there is a way to apply a resource only after the command initiated by Exec completes it's job. My requirement is that I should schedule a backup job only after another job is completed. My code, given below doesn't work. The cron gets scheduled as soon as the primary-backup-job is initiated.

Edit: I need to run primary job only once in life-time of the container(using docker). Primary backup job can take upto 5 hours and secondary around one hour. Primary-backup-script.py uses subprocess.popen module, to start the job , which forks the job in background, but I don't think that is important. Correct me if I am wrong.

exec{ 'primary-backup-job':
    command     => '/path/primary-backup-script.py',
    require     => File['/path/primary-backup-script.py'],
    refreshonly => true,
    }

# Schedule cron job only after primary backup is completed.
cron{ schedule-secondary-backup:
    command => "/path/secondary-backup-script.py",
    require => Exec['primary-backup-job'],
    user    => root,
    hour    => 05,
    minute  => 00,
    weekday => 1-5;
    }

Any thoughts or suggestions?

Is that `exec` resource forking to the background, or running as part of the puppet agent run? Also - how often is the primary job needing to be run, and does the secondary job always need to run immediately after the primary? I think to coordinate these effectively, you'll want to have either both as `exec` or both as `cron` resources, but which one works better depends on the needs of the job. — Shane Madden, Nov 23 '14 at 20:09

score 0 · Answer 1 · answered Nov 23 '14 at 21:47

Hmm.. Well, since it's running in the background, it's difficult to trigger something in response to it finishing. How would you feel about switching the secondary to something like this:

exec { 'secondary-backup':
    command => "/path/secondary-backup-script.py",
    unless  => "/bin/ps -ef | /bin/grep primary-backup-script",
}

..with the grep params set to something that'll find the primary backup process?

This'll make the secondary fire on the next Puppet run after the primary backup finishes.

score 0 · Accepted Answer · answered Nov 25 '14 at 00:29

You are using the tool incorrectly. Puppet is declarative language - meaning - you are supposed to declare states with puppet, and not orchestrate things. You can do it, offcourse, but it's harder, it will look clumsy and it will be error-prone.

When using puppet, one should avoid using 'exec' resource as much as possible. Why? Because exec is external to "state machine" - meaning exec will modify system in (for puppet) uncontrolled and unknown way.

Linking other resources to exec can even make matters worse.

Next issue is that puppet is not a cron job, so you should avoid running jobs via 'exec' from the Puppet itself. That's why we have cron.

In your specific case, wouldn't it be simpler to add cron job creation one-liner to the end of your original script? You would have multiple benefits:

no need to implement complex and error-prone hacks for messaging other process
no need to implement watchdog
no need to debug various issues that will come out from that solution.

If it's possible - just add cron creation at the end of your original first backup script. If it's proprietary tool or script - and you cannot edit it, just create simple bash wrapper, and call that wrapper via exec resource.

If you want to manage cron from puppet itself, you can still do that, but try to avoid linking exec and cron resource together.

What I would do in that case (where you want/must manage cron from puppet code) is:

cron { schedule-secondary-backup:
  command => "pgrep primary-backup-script && echo 'still running' || /path/secondary-backup-script.py",
  user    => root,
  hour    => 05,
  minute  => 00,
  weekday => 1-5;
}

Another solution is to modify primary script to create a file on filesystem when it finishes with execution, and the use following cron command in previous resource definition:

  command => "[ ! -e /path/to/first_script_finished_file ] && /path/secondary-backup-script.py",

Hope this helps.

Halt the puppet master while Exec is running a script

2 Answers2