3

I have a durable azure function (2.0) that calls back on itself for an eternal orchestration, based on this article. Code below. This works fine and runs well for approx 20 minutes (say around 50 iterations). However, after this time it goes to sleep and stops working. If I navigate to the Azure portal and just click on the functions app and click the functions list (i.e. I just click around but don't change any settings or stop/start anything) it suddenly "wakes up" and continues on no problem. There are no errors or issues in the logs, just a indication of the inactive period and then the logs resuming. Am I missing a setting somewhere?

public static async Task Run([OrchestrationTrigger] DurableOrchestrationContextBase ctx, ILogger log)
{
    try
    {
        var config = ctx.GetInput<Config>();
        config = await ctx.CallActivityAsync<Config>("GetConfig", null);
        await ctx.CallActivityAsync("ProcessDetails", config);

        var nextCheckpoint = ctx.CurrentUtcDateTime.AddSeconds(config.PollingIntervalInSeconds);

        await ctx.CreateTimer(nextCheckpoint, CancellationToken.None);             
        ctx.ContinueAsNew(config);   
    }
    catch (Exception ex)
    {
        log.LogError($"[Error!][Failed with: {ex.ToString()}]");
        throw;
    }
}

After hitting the refresh button as described by @davidebbo the issue still persists, see log below.

enter image description here

Further update, even though the issue on GitHub appears resolved, I still needed to do the double config setting as indicated below in order to get my function reliable.

{
  "version": "2.0",
  "extensions": {

    "durableTask": {
      "HubName": "myHub",
      "maxConcurrentActivityFunctions": 10,
      "maxConcurrentOrchestratorFunctions": 1
    }
  },
  "durableTask": {
    "HubName": "myHub",
    "maxConcurrentActivityFunctions": 10,
    "maxConcurrentOrchestratorFunctions": 1
  },
  "functionTimeout": "00:10:00"
} 

Now still having issues after much code optimisation, even though metrics look rock solid.

enter image description here

Murray Foxcroft
  • 12,785
  • 7
  • 58
  • 86

2 Answers2

3

This situation may happen when you are using the Consumption Plan for your Func. This means that after some period of infectivity azure will suspend your Func and when new event or request raised in a system it will reactivate your function. To solve this issue you can do:

  • use App Service plan
  • use Premium Plan
  • create some app to warm-up your func.

More details you can read in this article

Support Ukraine
  • 978
  • 6
  • 20
1

You were most likely running into this issue.

The issue is now fixed, but if you were affected, you need to explicitely sync the triggers once now that the fix is out. To do this, go to the Function App in the Portal and click the little refresh icon next to the Function App name in the tree. You just need to do this once.

Then you can leave the portal and it should keep happily running.

David Ebbo
  • 42,443
  • 8
  • 103
  • 117
  • This still happens on a fresh deploy of everything, but the work around of hitting refresh seems to be keeping it up for now. I'll try the w workaround hosts.json, a bit messy but seems like mine will be: { "extensions": { "version": "2.0", "durableTask": { "HubName": "myhub", "maxConcurrentActivityFunctions": 10, "maxConcurrentOrchestratorFunctions": 1 } }, "version": "2.0", "durableTask": { "HubName": "myhub", "maxConcurrentActivityFunctions": 10, "maxConcurrentOrchestratorFunctions": 1 }, "functionTimeout": "00:10:00" } – Murray Foxcroft Oct 08 '18 at 07:52
  • Hi David, unfortunately that sync does not work, the problem still persists. I have updated my question with the log / graph. I am now trying the duplicate config section as per the article you listed. – Murray Foxcroft Oct 08 '18 at 08:33
  • Ok, almost fixed with work-arounds. The duplicate hosts config is still required. New CI deployments however still require navigating to the functions instance and refreshing for them to take effect. – Murray Foxcroft Oct 08 '18 at 12:11
  • 1
    That's odd. Can you share the name of any test Function App that's in that state? – David Ebbo Oct 08 '18 at 16:07
  • Not publicly, no. – Murray Foxcroft Oct 10 '18 at 10:49
  • Still seeing random terminations too. Last night there a 500 on the entire functions app for no apparent reason. I rebuilt and redeployed everything, then it ran for a few more hours and went to sleep. Clicked the refresh on the functions blade and it started going again. It's very flaky. – Murray Foxcroft Oct 10 '18 at 11:58