Azure Data Factory's Web Hook Activity times out after 1 minute

Question

I have an Azure Data Factory V2 with a Web Hook Activity which is used to invoke an Azure Function with HTTP trigger, code as below.

using namespace System.Net

# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)

# Interact with query parameters or the body of the request.
$callBackUri = $Request.Body.callBackUri
Write-Output "CallBack url is : $callBackUri"

# Need to return Http 202 Accepted here 
# This is the issue, it does not actually return from this point at the moment
Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
    StatusCode = [HttpStatusCode]::Accepted
    Body = $body
}) -Clobber

# some long running processing
$seconds = 60
Write-Output "Returned http 202. Sleeping for $seconds seconds.."
Start-Sleep -Second $seconds
Write-Output "Sleep complete."

# Invoke the callback
Invoke-WebRequest -Uri $callBackUri -Method POST -ContentType 'application/json' -Body "This is a callback"

The Function is supposed to receive the HTTP request, return an HTTP 202 Accepted response immediately and then continue processing. After the process is complete it the function needs to invoke a POST on the callBackUri to indicate to the Web Hook Activity that the processing is complete.

However, the function does not return a 202, instead it completes it's long running process and then returns a 203. I do understand that initially output binding is set and it is returned only after the entire script is executed.

Is there a way around this? I am simply trying to implement this: https://mrpaulandrew.com/2019/06/18/azure-data-factory-web-hook-vs-web-activity/

score 0 · Accepted Answer · answered Aug 13 '20 at 04:25

I tried, Start-Job, Start-ThreadJob, Invoke-Command with all variations to attempt to invoke an async REST request without waiting (fire and forget) from the Function App but failed.

This seems logical to me because, if this was allowed people would simply run everything on background threads without waiting for them to complete and that would defeat the purpose of Function app - serverless.

What I've settled with is to use the same Invoke-RestRequest with a 1 second timeout, inside a try catch to suppress the timeout exception. This caused the request to immediately timeout and the Function app would complete.

The code looks like this:

$Body = @{
            callbackUri = $Request.Body.callBackUri;
        } | ConvertTo-Json
try{
    # This API will be responsible for issuing the call back after it has finished the long running process
    $output = Invoke-RestMethod -Method Post -Body $Body -Uri $funcapp2Url -ContentType 'application/json' -TimeoutSec 1
}
catch {
    Write-Output $_
}

# Return HTTP 202 immediately
Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
    StatusCode = [HttpStatusCode]::Accepted
    Body = "Wait for callback"
}) -Clobber

Azure Data Factory's Web Hook Activity times out after 1 minute

1 Answers1

Linked