Azure Service Bus Queue Trigger function is called more than once when deployed

Question

I have two Azure Functions. One is HTTP triggered, let's call it the API and the other one ServiceBusQueue triggered, and let's call this one the Listener.

The first one (the API) puts an HTTP request into a queue and the second one (the Listener) picks that up and processes that. The functions SDK version is: 3.0.7.

I have two projects in my solution for this. One which contains the Azure Functions and the other one which has the services. The API once triggered, calls a service from the other project that puts the message into the queue. And the Listener once received a message, calls a service from the service project to process the message.

Any long-running process?

The Listener actually performs a lightweight workflow and it all happens very quickly considering the amount of work it executes. The average time of execution is 90 seconds.

What's the queue specs?

The queue that the Listener listens to and is hosted in an Azure ServiceBus namespace has the following properties set:

• Max Delivery Count: 1
• Message time to live: 1 day
• Auto-delete: Never
• Duplicate detection window: 10 min
• Message lock duration: 5 min

And here a screenshot for it:

The API puts the HTTP request into the queue using the following method:

 public async Task ProduceAsync(string queueName, string jsonMessage)
 {
        jsonMessage.NotNull();
        queueName.NotNull();

        IQueueClient client = new QueueClient(Environment.GetEnvironmentVariable("ServiceBusConnectionString"), queueName, ReceiveMode.PeekLock)
        {
            OperationTimeout = TimeSpan.FromMinutes(5)
        };

        await client.SendAsync(new Message(Encoding.UTF8.GetBytes(jsonMessage)));

        if (!client.IsClosedOrClosing)
        {
            await client.CloseAsync();
        }
 }

And the Listener (the service bus queue triggered azure function), has the following code to process the message:

[FunctionName(nameof(UpdateBookingCalendarListenerFunction))]
public async Task Run([ServiceBusTrigger(ServiceBusConstants.UpdateBookingQueue, Connection = ServiceBusConstants.ConnectionStringKey)] string message)
{
        var data = JsonConvert.DeserializeObject<UpdateBookingCalendarRequest>(message);
        _telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarListenerFunction)} picked up a message at {DateTime.Now}. Data: {data}");

        await _workflowHandler.HandleAsync(data);
}

The Problem

The Listener function processes the same message 3 times! And I have no idea why! I've Googled and read through a few of StackOverFlow threads such as this one. And it looks like that everybody advising to ensure lock duration is long enough for the process to get executed completely. Although, I've put in 5 minutes for the lock, yet, the problem keeps coming. I'd really appreciate any help on this.

Answer 1

Just adding this in here so might be helpful for some others.

After some more investigations I've realized that in my particular case, the issue was regardless of the Azure Functions and Service Bus. In my workflow handler that the UpdateBookingCalendarListenerFunction sends messages to, I was trying to call some external APIs in a parallel approach, but, for some unknown reasons (to me) the handler code was calling off the external APIs one additional time, regardless of how many records it iterates over. The below code shows how I had implemented the parallel API calls and the other code shows how I've done it one by one that eventually led to a resolution for the issue I had.

My original code - calling APIs in parallel

 public async Task<IEnumerable<StaffMemberGraphApiResponse>> AddAdminsAsync(IEnumerable<UpdateStaffMember> admins, string bookingId)
    {
        var apiResults = new List<StaffMemberGraphApiResponse>();

        var adminsToAdd = admins.Where(ad => ad.Action == "add");

        _telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarWorkflowDetailHandler)} Recognized {adminsToAdd.Count()} admins to add to booking with id: {bookingId}");

        var addAdminsTasks = adminsToAdd.Select(admin => _addStaffGraphApiHandler.HandleAsync(new AddStaffToBookingGraphApiRequest
        {
            BookingId = bookingId,
            DisplayName = admin.DisplayName,
            EmailAddress = admin.EmailAddress,
            Role = StaffMemberAllowedRoles.Admin
        }));

        if (addAdminsTasks.Any())
        {
            var addAdminsTasksResults = await Task.WhenAll(addAdminsTasks);
            apiResults = _populateUpdateStaffMemberResponse.Populate(addAdminsTasksResults, StaffMemberAllowedRoles.Admin).ToList();
        }

        return apiResults;
    }

And my new code without aggregating the API calls into the addAdminsTasks object and hence with no await Task.WhenAll(addAdminsTasks):

 public async Task<IEnumerable<StaffMemberGraphApiResponse>> AddStaffMembersAsync(IEnumerable<UpdateStaffMember> members, string bookingId, string targetRole)
    {
        var apiResults = new List<StaffMemberGraphApiResponse>();

        foreach (var item in members.Where(v => v.Action == "add"))
        {
            _telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarWorkflowDetailHandler)} Adding {targetRole} to booking: {bookingId}. data: {JsonConvert.SerializeObject(item)}");

            apiResults.Add(_populateUpdateStaffMemberResponse.PopulateAsSingleItem(await _addStaffGraphApiHandler.HandleAsync(new AddStaffToBookingGraphApiRequest
            {
                BookingId = bookingId,
                DisplayName = item.DisplayName,
                EmailAddress = item.EmailAddress,
                Role = targetRole
            }), targetRole));
        }

        return apiResults;
    }

I've investigated the first approach and the numbers of tasks were exact match of the number of the IEnumerable input, yet, the API was called one additional time. And within the _addStaffGraphApiHandler.HandleAsync, there is literally nothing than an HttpClient object that raises a POSTrequest. Anyway, using the second code has resolved the issue.