5

Subject says it all really :) Say I've got a pretty busy Azure continuous webjob that is processing from an azure Queue:

    public static void ProcessQueue([QueueTrigger("trigger")] Info info)
    { .... }

If I re-deploy the webjob, I can see that any currently executing job seems to be aborted (I see a "Never Finished" status). Is that job replayed after I release or is it lost forever?

Also, is there a nice way to make sure that no jobs are running when we deploy webjobs, or is it up to the developer to code a solution to that (such as a config flag that is checked every run).

Thanks

Matt Roberts
  • 26,371
  • 31
  • 103
  • 180

1 Answers1

7

When a WebJob that uses the WebJobs SDK picks up a message from a queue, it acquires it with a 10 minutes lease. If the job process dies while processing the message, the lease expires after 10 minutes and the message goes back in the queue. If the WebJob is restarted, it will pick that message again. The message is only deleted if the function completes successfully.

Therefore, if the job dies and restarts immediately, like in the case of a redeploy, it might take up to 10 minutes to pick again the message. Also, because of this, it is recommended to either save state yourself or make the function idempotent.

In the WebJobs Dashboard you will see two invocations for the same message. One of them will be marked as Never Finished because the function execution never completed.

Unfortunately, there is no out of the box solution to prevent jobs from running during deploy. You would have to create your own logic that notifies (through a queue message?) that a deploy is about the start and then aborts the host. The host abort will wait for any existing function to stop and will prevent new ones from starting. However, this is a very tricky situation if you have multiple instances of the webjob because only one of them will get the notification.

Victor Hurdugaci
  • 28,177
  • 5
  • 87
  • 103
  • Thanks @Victor. A simpler solution I was thinking about was just manually stopping the webjob before I re-deploy. But I can't find any documentation that tells me if stopping a webjob will do so "gracefully". I'll have to experiment with that – Matt Roberts Jan 26 '15 at 11:36
  • If you stop the webjob, we take care of the [shutdown notification marker](http://blog.amitapple.com/post/2014/05/webjobs-graceful-shutdown/#.VMZkqY0tGM8) file and gracefully shutdown, assuming your functions completes in the maximum allocated time (I think it is 1 min by default) – Victor Hurdugaci Jan 26 '15 at 16:01