Stop Hangfire starting a recurring job if it's already running

Question

I have a list of recurring jobs set up in Hangfire, all with an interval of 20 minutes.

They all call the same method, but with different parameters. e.g.

Job ID test_1 => MyTestMethod(1)
Job ID test_50 => MyTestMethod(50)
Job ID test_37 => MyTestMethod(37)

Sometimes, a job might take longer than its interval, i.e. longer than 20 minutes, and I want to make sure that job doesn't get run again whilst another instance of it is already running.

The DisableConcurrentExecutionAttribute is not suitable here, because the method CAN be run concurrently, just not with the same parameter(s).

I attempted to have a static Dictionary<string, DateTime> that tracked whether a job was already running so that it could block concurrent method calls, but the problem there is that if the application restarts for any reason, then it "forgets" what jobs are currently running (Hangfire of course will continue to run in background).

UPDATE
I also tried adding a JobState table to my database to track which jobs are running and then checking that table when MyTestMethod starts, but this relies on MyTestMethod setting running=1 when it starts and running=0 when it ends and if the thread crashes halfway through that job (for whatever reason) that might not happen, thus preventing the job running again at all.

I am sure I can solve this by simply querying Hangfire job status by job ID - I just can't find how to do this?

Check this out: http://docs.hangfire.io/en/latest/background-processing/tracking-progress.html (I suggest you look at the Highlighter example to understand the context of the example). — aethercowboy, Jun 09 '16 at 14:31

score 9 · Answer 1 · answered Dec 22 '16 at 15:03

I had a similar requirement, basically I wanted to use the DisableConcurrentExecutionAttribute but have it take into account the parameters. That way if a job gets queued with the same parameter it will still run just not in parallel. I took the example DisableMultipleQueuedItemsFilter which actually removes jobs and modified the DisableConcurrentExecutionAttribute to use the parameters. The difference is the jobs will be queued if they have the same parameters list they won't run in parallel.

A full example can be seen here with both attributes: https://gist.github.com/sbosell/3831f5bb893b20e82c72467baf8aefea

The relevant code for the attribute:

 public class DisableConcurrentExecutionWithParametersAttribute : JobFilterAttribute, IServerFilter
    {
        private readonly int _timeoutInSeconds;

        public DisableConcurrentExecutionWithParametersAttribute (int timeoutInSeconds)
        {
            if (timeoutInSeconds < 0) throw new ArgumentException("Timeout argument value should be greater that zero.");

            _timeoutInSeconds = timeoutInSeconds;
        }

        public void OnPerforming(PerformingContext filterContext)
        {
            var resource = GetResource(filterContext.BackgroundJob.Job);

            var timeout = TimeSpan.FromSeconds(_timeoutInSeconds);

            var distributedLock = filterContext.Connection.AcquireDistributedLock(resource, timeout);
            filterContext.Items["DistributedLock"] = distributedLock;
        }

        public void OnPerformed(PerformedContext filterContext)
        {
            if (!filterContext.Items.ContainsKey("DistributedLock"))
            {
                throw new InvalidOperationException("Can not release a distributed lock: it was not acquired.");
            }

            var distributedLock = (IDisposable)filterContext.Items["DistributedLock"];
            distributedLock.Dispose();
        }

        private static string GetFingerprint(Job job)
        {
            var parameters = string.Empty;
            if (job?.Arguments != null)
            {
                parameters = string.Join(".", job.Arguments);
            }
            if (job?.Type == null || job.Method == null)
            {
                return string.Empty;
            }
            var payload = $"{job.Type.FullName}.{job.Method.Name}.{parameters}";
            var hash = SHA256.Create().ComputeHash(System.Text.Encoding.UTF8.GetBytes(payload));
            var fingerprint = Convert.ToBase64String(hash);
            return fingerprint;
        }
        private static string GetResource(Job job)
        {
            return GetFingerprint(job);
        }
    }

score 5 · Answer 2 · answered Jun 08 '16 at 19:29

5

There's an attribute you can use to prevent concurrent execution (for a time):

[DisableConcurrentExecution(3600)] // argument in seconds, e.g., an hour
public void MyTestMethod(int id) {
  // ...
}

answered Jun 08 '16 at 19:29

aethercowboy

444
4
16

1

So far as I know this only works for the method irrespective of its parameters e.g. the id passed to the method. I will need to run the same method concurrently, just not with the same parameter having been passed to it (i hope that makes sense) – Jimbo Jun 09 '16 at 06:37
2

Good point. Didn't notice you said that. There are a few additional things to try, one being a filter: https://gist.github.com/odinserj/a8332a3f486773baa009 and check out the suggestions here: https://discuss.hangfire.io/t/checking-for-a-job-state/57/7 – aethercowboy Jun 09 '16 at 14:41
1

Brilliant, DisableMultipleQueuedItemsFilter attribute looks to do exactly what I need - will check it out then you can post that as an answer and score some points :P – Jimbo Jun 10 '16 at 06:35
Thanks, but I'll let you figure it out so you can post your own answer. :) – aethercowboy Jun 10 '16 at 14:11

score 0 · Answer 3 · answered May 12 '23 at 15:56

Try a JobFilter like this: SkipWhenPreviousJobIsRunningAttribute.cs. There isn't great documentation on it, but it's referenced from Skip the Job Execution if previous instance is still running, Not launching a scheduled job if the previous one didn't complete, and How to disable Concurrent execution of long running hangfire job?.

Stop Hangfire starting a recurring job if it's already running

3 Answers3