0

I have a durable function which doesn't appear to crash but after first invocation just keeps executing the same function. After this first invocation trying to set breakpoints has no effect.

[30/11/2017 16:16:21] Function started (Id=972ee93c-ab61-4834-937c-207e8953821d)
[30/11/2017 16:16:21] Executing 'CompileFeatureObservations' (Reason='', Id=972ee93c-ab61-4834-937c-207e8953821d)
[30/11/2017 16:16:21] Starting Feature Compilation.
[30/11/2017 16:16:21] Function completed (Success, Id=972ee93c-ab61-4834-937c-207e8953821d, Duration=58ms)
[30/11/2017 16:16:21] Executed 'CompileFeatureObservations' (Succeeded, Id=972ee93c-ab61-4834-937c-207e8953821d)
[30/11/2017 16:16:21] b540b650019244719a7f3a61e45735f4: Function 'CompileFeatureObservations (Activity)', version '' completed. ContinuedAsNew: False. IsReplay: False. Output: (62123 bytes). State: Completed. HubName: DurableFunctionsHub. AppName: . SlotName: . ExtensionVersion: 1.0.0.0.

The only factor that I can see seems to have influence is the size of the request payload although it is below the 65kb limit.

It is using the fan-out/fan-in pattern as described in the documentation. When the size of my task array gets to ~100 it seems to stop working and then go into the endless cycle.

Perhaps I have exceeded the fan-out limit? Is there a way to control the number of function "instances" spun up?

I'm using consumption plan.

The only way I can find to stop the behaviour is to stop the local storage emulator and delete and re-init the underlying localdb.

Does anyone have suggestions as to further troubleshoot?

alwayslearning
  • 4,493
  • 6
  • 35
  • 47
phil
  • 1,938
  • 4
  • 23
  • 33
  • You should share your code – Mikhail Shilkov Nov 30 '17 at 16:21
  • I don't have permission. It is running a map/reduce like computation. During debug no errors and try/catch blocks are not catching exceptions. I definitely think its linked to the number of fan-out ops. – phil Nov 30 '17 at 16:41

1 Answers1

3

I'm pretty confident that the problem is the large output of your function, which is a known issue mentioned here: https://github.com/Azure/azure-functions-durable-extension/issues/79.

The thing which is misleading here is this log statement:

[30/11/2017 16:16:21] b540b650019244719a7f3a61e45735f4: Function 'CompileFeatureObservations (Activity)', version '' completed. ContinuedAsNew: False. IsReplay: False. Output: (62123 bytes). State: Completed.

It claims the output is about 60 KB, but the number reported here is actually not correct, since it assumes UTF-8 encoding. In reality, Azure Storage uses UTF-32 encoding, so the real size is probably much larger than this. I'll make a note that this needs to be fixed. In the beta2 update, we will report this correctly and throw an exception. Sometime after that, we'll have support for arbitrarily large return values.

Just to answer the other parts of your question, the degree of fan-out is not the issue. It's simply the size of your return value. If you can shrink this down, then that should fix your problem.

Chris Gillum
  • 14,526
  • 5
  • 48
  • 61
  • Ah, that explains why my input payload was also reported as too large even though it was under 60KB too. – phil Dec 01 '17 at 10:27
  • So I thought the way to go here was to save my result of `CompileFeatureObservations` function to documentdb and then retrieve in my orchestrator but it looks like your not allowed to do that in the orchestrator. So not sure how to then compute the fan-out and initiate from the orchestrator. In a nutshell how to pass around large-ish payloads. – phil Dec 01 '17 at 18:07
  • 1
    We're working on large message support (https://github.com/Azure/azure-functions-durable-extension/issues/26). Until that happens, you'll need to refactor your orchestration. One option is to do smaller and/or nested fan-outs/fan-ins using sub-orchestrations. If you need to pass around large-ish payloads, you can also save data into blob storage using the blob storage bindings and pass around the URLs. – Chris Gillum Dec 01 '17 at 22:51