4

When Azure WebJobs are stopped or restarted for whatever reason, there is a default grace period of 5 seconds before the host console process is terminated. Apparently it is possible to increase this time period by including a "settings.job" file with the WebJob when it is published to Azure, as explained here. Unfortunately I cannot get this to work for my website, and the shutdown period is always 5 seconds, despite this being a continuous WebJob that is published to a Basic App Service that is set to "Always On".

As per the sample code from the link, my Program.cs looks like this:

class Program
{
    static void Main()
    {
        var host = new JobHost();
        host.RunAndBlock();
    }
}

and my Functions.cs looks like this:

public class Functions
{
    public static async Task ProcessQueueMessageAsync(
        [QueueTrigger("testq")] string message, TextWriter log,
        CancellationToken cancellationToken)
    {
        Console.WriteLine("Function started.");
        try
        {
            await Task.Delay(TimeSpan.FromMinutes(10), cancellationToken);
        }
        catch (TaskCanceledException)
        {
            Shutdown().Wait();
        }
        Console.WriteLine("Function completed succesfully.");
    }

    private static async Task Shutdown()
    {
        Console.WriteLine("Function has been cancelled. Performing cleanup ...");
        await Task.Delay(TimeSpan.FromSeconds(30));
        Console.WriteLine("Function was cancelled and has terminated gracefully.");
    }
}

I have added a settings.job file to the project, and in its Properties in Visual Studio I have set it to "Copy always". The content of the file is as follows:

{ "stopping_wait_time": 60 }

The WebJob is published with the website. By using the Kudu tools and going to the debug console I can verify that the "settings.job" file is being copied across to the same location as the WebJob host executable:

Kudu screenshot

However, if I look at the link https://xxx.scm.azurewebsites.net/api/continuouswebjobs the returned JSON does not include any settings at all:

{
    "status":"Stopped",
    "detailed_status":"f9e13c - Stopped\r\n",
    "log_url":"https://...",
    "name":"WebJobTest",
    "run_command":"WebJobTest.exe",
    "url":"https://...",
    "extra_info_url":"https://...",
    "type":"continuous",
    "error":null,
    "using_sdk":true,
    "settings":{
    }
}

Consequently, when I stop the WebJob from the Azure portal I end up with something like this in the logs:

[04/18/2016 12:53:12 > f9e13c: SYS INFO] Run script 'WebJobTest.exe' with script host - 'WindowsScriptHost'
[04/18/2016 12:53:12 > f9e13c: SYS INFO] Status changed to Running
[04/18/2016 12:53:13 > f9e13c: INFO] Found the following functions:
[04/18/2016 12:53:13 > f9e13c: INFO] WebJobTest.Functions.ProcessQueueMessageAsync
[04/18/2016 12:53:13 > f9e13c: INFO] Job host started
[04/18/2016 13:01:58 > f9e13c: INFO] Executing: 'Functions.ProcessQueueMessageAsync' - Reason: 'New queue message detected on 'testq'.'
[04/18/2016 13:01:58 > f9e13c: INFO] Function started.
[04/18/2016 13:02:47 > f9e13c: SYS INFO] Status changed to Disabling
[04/18/2016 13:02:53 > f9e13c: SYS INFO] Detected WebJob file/s were updated, refreshing WebJob
[04/18/2016 13:02:53 > f9e13c: SYS INFO] Status changed to Stopping
[04/18/2016 13:02:53 > f9e13c: INFO] Function has been cancelled. Performing cleanup ...
[04/18/2016 13:02:58 > f9e13c: ERR ] Thread was being aborted.
[04/18/2016 13:02:58 > f9e13c: SYS INFO] WebJob process was aborted
[04/18/2016 13:02:58 > f9e13c: SYS INFO] Status changed to Stopped

Note the standard 5 second gap between "Status changed to Stopping" and "Thread was being aborted".

Why is the "settings.job" file being ignored?

Steven Rands
  • 5,160
  • 3
  • 27
  • 56
  • For continuous jobs, I think it works differently. You need to look for WEBJOBS_SHUTDOWN_FILE http://blog.amitapple.com/post/2014/05/webjobs-graceful-shutdown/#.VxTwkI9OJPY – Mehmet Aras Apr 18 '16 at 14:36
  • Possibly a duplicate of http://stackoverflow.com/questions/34543086/continuous-webjob-stopping-wait-time-ignored – David Ebbo Apr 18 '16 at 17:52
  • verify that the `settings.job` file is in the root of the webjob directory, you can browse to https://{sitename}.scm.azurewebsites.net/debugconsole and there to directory: `d:\home\site\wwwroot\app_data\jobs\continuous\{jobname}` – Amit Apple Apr 18 '16 at 22:20
  • should be { "stopping_wait_time": 60, "is_singleton": true } in settings.job – Alex Chen-WX Apr 19 '16 at 06:58
  • @DavidEbbo Similar yes, however on the other question the "stopping_wait_time" value was being detected by the Azure infrastructure whereas my value seems to be ignored. – Steven Rands Apr 19 '16 at 08:35
  • @MehmetAras I don't think that is the case. Looking at the logs I can see "Function has been cancelled. Performing cleanup ..." appears: the issue is that I'm not being given long enough to cancel properly, not that cancellation is never detected. – Steven Rands Apr 19 '16 at 08:38
  • @AlexChen-WX Why would the "is_singleton" property make a difference? Surely that just ensures that I only end up with a single instance of the WebJob, and wouldn't affect how long the host waits for cancellation. – Steven Rands Apr 19 '16 at 08:43
  • @AmitApple Yes, I have verified that the "settings.job" file is being copied to that location on the server. I even opened the file just to check that it wasn't empty: it had the correct contents. Just seems like the Azure WebJob infrastructure isn't reading it. – Steven Rands Apr 19 '16 at 08:54
  • Can you share your web app name, either directly or [indirectly](https://github.com/projectkudu/kudu/wiki/Reporting-your-site-name-without-posting-it-publicly)? This will help us investigate. Thanks. – David Ebbo Apr 19 '16 at 13:59

1 Answers1

8

The problem is that your settings.job file has two sets of UTF-8 BOM, which is highly unusual and throws off the parser.

Specifically, it has the following sequence at the beginning (view using a hex editor): EF BB BF EF BB BF. Normally, the UTF-8 BOM is just EF BB BF.

What editor are you using that caused that?

In any case, once you fix that up, it should work fine.

David Ebbo
  • 42,443
  • 8
  • 103
  • 117
  • That was it! I changed the file to remove the duplicate BOMs, redeployed the web app, and it worked as intended! Thanks very much, I never would have found that by myself. – Steven Rands Apr 20 '16 at 09:01
  • As to how the dupe BOMs ended-up in the file in the first place: I added a new item to the project in VS (right click, Add > New Item), selected "Text File" and changed the name to "settings.job". Then I copied the contents of the file from [here](https://github.com/mathewc/samples/blob/master/WebJobSamples/ContinuousJobGracefulShutdown/settings.job), pasted them into the file, then saved it. This gives me the dupe BOMs. I think it must be a peculiarity of the web browser I'm using (Chrome) in that sometimes I get dupe BOMs when I paste the text into the file, and sometimes I don't. – Steven Rands Apr 20 '16 at 09:11
  • Strange. I wish the editor made it more obvious that there is something funky about it. notepad actually behaved in a very peculiar way with that file, where it was showing some text in a different size, and editing was messed up. – David Ebbo Apr 20 '16 at 16:55
  • Someone else ran into same [here](https://social.msdn.microsoft.com/Forums/azure/en-US/764c7776-ca87-4246-a1ea-f6253efd498b/cronscheduled-webjob-not-working?forum=windowsazurewebsitespreview). – David Ebbo Apr 21 '16 at 15:02
  • When you say you copied the content of the file from [here](https://github.com/mathewc/samples/blob/master/WebJobSamples/ContinuousJobGracefulShutdown/settings.job), do you mean that you selected the text and hit Copy directly in the browser, or that you had cloned that repo and then copied the content on your local machine? – David Ebbo Apr 21 '16 at 15:04
  • Never mind, I just reproed it by copying straight from that page. Very strange. I'll try to figure out if it's a VS bug. – David Ebbo Apr 21 '16 at 15:06
  • Yeah, I just selected the text with the mouse in Chrome, copied with Ctrl+C, then switched to VS and pasted with Ctrl+V. FWIW I'm using VS 2013 on Windows 7. As I said in the earlier comment it doesn't insert the dupe BOMs every time though. – Steven Rands Apr 21 '16 at 15:11
  • I actually reproed it pasting in notepad as well, so we can't blame VS. I'd say it's actually a Chrome issue. When you copy from Chrome, if you start **exactly** on the open brace, it doesn't happen. But if you start selecting a little bit to the left of it, then it includes the BOM in the clipboard, causing the issue. In IE, I couldn't repro. – David Ebbo Apr 21 '16 at 18:24
  • 1
    For those with this problem (I know, 2019). I didn't had the BOM issues, but I had spaces (I also copied from GitHub). So, this `{ "stopping_wait_time": 60 }` didn't work. This `{"stopping_wait_time":60}` worked. You can use the `https://xxx.scm.azurewebsites.net/api/continuouswebjobs` to check. After the change, I see the values there. – jpgrassi Sep 06 '19 at 12:37