1

A strange this is happening in one of my WebJobs and I cannot find what's wrong.

This WebJob is configured to use ServiceBusTrigger as soon as a message is received on a particular Service Bus Queue. This function has an execution time of minutes / hours, so I configured the JobHostConfiguration like this:

config.UseServiceBus(new ServiceBusConfiguration()
{
     MessageOptions = new Microsoft.ServiceBus.Messaging.OnMessageOptions()
     {
          MaxConcurrentCalls = 3,
          AutoRenewTimeout = new TimeSpan(4, 0, 0),
          AutoComplete = true                    
     }
});

If I understood correctly, this would grant that each function:

  1. Is executed maximum three times per instance in parallel (one per each message)
  2. The lock, on the message, is kept for a maximum of 4 hours. This would avoid the messages to be requeued if the processing time is less than 4 hours.
  3. As soon as the function completes, the message is removed from the queue (even if it should be the default behavior).

These messages contains jobs that must be executed for different users. In order to be executed in parallalel for different users, but as a singleton for each user, inside the function I take a lease (different from each user) and I keep it for 60 seconds, renewing it every 30 seconds with a Task.

This seems to work: jobs are executed in parallel for different customer and same requests for same users are serialized.

Now, sometimes it happens that the WebJob stops triggering the function, even if the queue is full of messages. I don't understand why.

As soon as I restart that webjob, everything runs as before. Any ideas on why this happens?

What I noticed is that the App was (wrongly) stopped, even if the webjobs were running. Maybe that was the problem?

Marconline
  • 1,108
  • 12
  • 30
  • Could you check the webjob logs as in here: https://stackoverflow.com/q/32844626. Judging from the logs there, the website shutting down would also stop the webjob. – juunas Jan 19 '18 at 10:26
  • I discovered that under the Dashboard the function was marked with a yellow exclamation mark with "Host not running" message. What does it mean?! – Marconline Jan 19 '18 at 14:38
  • I guess that means the process that runs the continuous WebJob is not running – juunas Jan 19 '18 at 15:02
  • Any ideas on why should it stop? – Marconline Jan 19 '18 at 15:05
  • Hmm, does your app service have Always On enabled? – juunas Jan 19 '18 at 15:06
  • Yes. The strange thing is that in the same app there are multiple webjobs (> 10) and all of them, except this one, run correctly. I cannot understand what's wrong. – Marconline Jan 19 '18 at 15:27
  • Maybe this is the reason! https://github.com/Azure/azure-webjobs-sdk-script/issues/2243 – Marconline Jan 19 '18 at 15:39
  • 1
    That is possible – juunas Jan 19 '18 at 15:40
  • 1
    Thanks for your help! I'm modifying my application specifying a longer timeout (using the Timeout attribute) and avoiding the ThrowOnTimeout. If this will work, I'll write the answer here. Thanks again – Marconline Jan 19 '18 at 15:43
  • Nope, that didn't change... Trying to figure out what's happening here... – Marconline Jan 22 '18 at 09:07
  • Anybody ever found a solution to this? – Jakob May 12 '20 at 10:28
  • 1
    No, I'm still facing the issue. Even right now. I feel it's related to a memory outage. – Marconline May 13 '20 at 15:36
  • Not sure if it helps anyone, but in our case we sometimes had a deadlock in the application, and Webjobs don't time out if an operation takes too long. Specifying timeouts with CancellationTokens in the code solved the issue. – Jakob Jul 06 '20 at 10:19

0 Answers0