How to schedule Sidekiq jobs according to a recurring schedule in database?

Question

I've set up a system where Measurements are run based on MeasurementSettings. Each setting defines a schedule using ice_cube (and some other properties for measurements). I'm able to edit this schedule for each setting and then poll for the next occurrences with something like:

def next_occurrences
  schedule.occurrences(Time.now + 1.day)
end

This gives me a set of timestamps when there should be a Measurement.

Now, I also have Sidekiq installed, and I can successfully run a measurement at a specific time using a MeasurementWorker. To do that, I just create an empty Measurement record, associate it with its settings, and then perform_async (or perform_at(...)) this worker:

class MeasurementWorker
  include Sidekiq::Worker
  sidekiq_options retry: 1, backtrace: true

  def perform(measurement_id)
    Measurement.find(measurement_id).run
  end
end

What's missing is that I somehow need to create the empty measurements based on a setting's schedule. But how do I do this?

Say I have the following data:

MeasurementSetting with ID 1, with a daily schedule at 12:00 PM
MeasurementSetting with ID 2, with an hourly schedule at every full hour (12:00 AM, 01:00 AM, etc.)

Now I need to:

Create a Measurement that uses setting 1, every day at 12:00
Create a Measurement that uses setting 2, every full hour
Call the worker for these measurements

But how?

Should I check, say every minute, whether there is a MeasurementSetting that is defined to occur now, and then create the empty Measurement records, then run them with the specific setting using Sidekiq?

Or should I generate the empty Measurement records with their settings in advance, and schedule them this way?

What would be the simplest way to achieve this?

score 0 · Answer 1 · answered Aug 13 '14 at 16:32

0

I would use cron every minute to look for MeasurementSettings that should run NOW. If yes, create a Sidekiq job to run immediately which populates the Measurement.

answered Aug 13 '14 at 16:32

Mike Perham

21,300
6
59
61

I guess that `whenever` would come in handy here. But how would I time it, like… when I have cron run at, say, 12:00 PM, then check if the setting is `.occurring_at(Time.now)`, I'd see that it should run, and start the measurement. But at 12:01 PM, the same measurement would still be running. Is there a way to catch this before using some interaction between `ice_cube` and Sidekiq or should I check if there is a measurement with this exact setting already running myself? – slhck Aug 13 '14 at 18:11
Preventing Measurements from stacking up is a common problem with scheduled jobs. There is no easy solution. – Mike Perham Aug 13 '14 at 22:46

score -1 · Accepted Answer · answered Dec 12 '14 at 07:05

Here's how I successfully did it:

Update your model that should run on a scheduled basis with a field planned_start_time. This will hold the time when it was planned to start at.
Use the whenever Gem to run a class method every minute, e.g. Measurement.run_all_scheduled

In that method, go through each setting (i.e. where the schedule is), and check if it is occurring now:

setting = MeasurementSetting.find(1) # get some setting, choose whatever
schedule = setting.schedule
if not schedule.occurring_at? Time.now
  # skip this measurement, since it's not planned to run now

If that's the case, then check if we can run the measurement by looking in the database if there isn't any previous measurement with the same planned start time. So first, we have to get the planned start time for the current schedule, e.g. when it's now 3:04 PM. the planned start time could have been 3:00 PM.
```
this_planned_start_time = schedule.previous_occurrence(Time.now).start_time
```
Then we check if the last measurement's start time (limit(1) gets just the last one) is the same or not.
```
if Measurement.limit(1).last.planned_start_time != this_planned_start_time
  # skip this one, since it was already run
```

If not, we can continue setting up a measurement.

measurement = Measurement.create measurement_setting: setting, 
                                 planed_start_time: this_planned_start_time

Then run it:
```
measurement.delay.run
```

How to schedule Sidekiq jobs according to a recurring schedule in database?

2 Answers2