Queue triggered based function app not getting completed

Question

We are using queue trigger based function app on premium plan where messages contains some details like azure subscriptions name. Based on which for each subscription we do many api calls specially to azure storage accounts(around 400 to 500). Since 'list' api call to storage account is limited to 100 call/5min, we get 429 response error on 101th call. To mitigate this we have applied exponential retry logic(tried both our own or polly library) which call after certain delay of time. This works for some subscription but fails for many where the retry logic does not try after first trying(we kept 3 retries with 60 sec delay). Even while monitoring the function app through live metrics we observed that sometimes cpu usage of some function instance goes to zero(although we do some operation like logging or use for loop in delay operation so that the function can be alive) which leads to killing of that particular function instance and pushing the message back to queue and start the process again with a fresh instance.

Note that since many subscription are processed in parallel, function app automatically scale up as required. Also since we are using premium plan one VM is always on state. So killing of any instance(which call around 400 to 500 storage api call for any particular subscription) is weird since in our delay the thread sleep time is only 10 sec for around 6,12,18(Time_delay) iteration. The below delay function is used in our retry logic code.

private void Delay(int Time_delay, string requestUri, int retryCount)
{
   for (int i = 0; i < Time_delay; i++)
   {
       _logger.LogWarning($"Sleep initiated for id: {requestUri.ToString()}, RetryCount: {retryCount} CurrentTimeDelay: {Time_delay}");
        Thread.Sleep(10000);
        _logger.LogWarning($"Sleep completed for id: {requestUri.ToString()}, RetryCount: {retryCount} CurrentTimeDelay: {Time_delay}");
    }
}

Note** Function app is not throwing any other exception other than dependency of 429 error response.

score 0 · Answer 1 · answered Sep 25 '20 at 18:48

Would it be possible for you to requeue instead of using Thread.Sleep? You can use initial visibility delay when requeuing:

public class Function1
{
    [FunctionName(nameof(TryDoWork))]
    public static async Task TryDoWork(
        [QueueTrigger("some-queue")] SomeItem item,
        [Queue("some-queue")] CloudQueue queue)
    {
        var result = _SomeService.SomeWork(item);
            
        if (result == 429)
        {
            item.Retries++;
            var json = JsonConvert.SerializeObject(item);
            var message = new CloudQueueMessage(json);
            var delay = TimeSpan.FromSeconds(item.Retries);
            await queue.AddMessageAsync(message, null, delay, null, null);
        }
    }
}

It might be that the sleeping is causing some wonky function app behavior. I think I remember reading some issues pertaining to the usage of Thread.Sleep, but I can't find it right now.

Also, you might want to add some sort of handling of messages that end up retrying more than 3 times (or however many you think is reasonable).

Queue triggered based function app not getting completed

1 Answers1