Azure service bus deadletter queue message unable to stay in deadletter for longer time

Question

I have Azure Main function app which processes coming messages from service bus.

If exception occurs because of API is down then I push message into Dead Letter Queue.

I have another azure Dead function app which continuously read DLQ and push back message into main queue.

And main Function app check if api is down then push message again to DLQ this should go on until API is up and Main function app process message successfully.

But issue is after some time message purge automatically.

I check Default Message Time To Live is 2 hours 30 minutes but message unable to stay there so long. I think in Dead Function app when I push message back to Main. in main queue message may be identified as duplicate but not sure.

What should I do to sustain message into DLQ for given Default Message Time To Live?

Dead letter function app code which push back message to main queue -

 MessageSender sender = new MessageSender(Environment.GetEnvironmentVariable("ConnectionStringSettingName"), UtilityHelper.GetAndValidateAppSetting("TopicName"), RetryPolicy.Default);

            Message deadLetterMessage = new Message(message.Body);

            foreach (KeyValuePair<string, object> userProperty in message.UserProperties)
            {
                deadLetterMessage.UserProperties.Add(userProperty.Key, userProperty.Value);
            }

            //Send the message to the Active Queue
            await sender.SendAsync(deadLetterMessage);

Main Function App Code-

[FunctionName("ProcessMessage")]
        public static async System.Threading.Tasks.Task RunAsync([ServiceBusTrigger("mytopic", "mysub", Connection = "ConnectionStringSettingName")]Message mySbMsg, string lockToken, MessageReceiver messageReceiver, ILogger log)
        {
            
            try
            { 
                  log.LogInformation("Process message...");
            }
            catch (Exception ex)
            {
                log.LogInformation($"Send message to dead-letter: {ex}");
                await messageReceiver.DeadLetterAsync(lockToken);
            }
        }

My logic is: until API is up I'm sending message into DLQ and DLQ function app pushing it back to main queue so that I don't lost message.

Update

Find out root cause:

Microsoft.Azure.ServiceBus.ServiceBusCommunicationException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full ErrorCode: NoBufferSpaceAvailable ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was fullat Microsoft.Azure.ServiceBus.ServiceBusConnection.CreateConnectionAsync(TimeSpan timeout)

How to tackle this?

This question prior to editing was in a dreadfully unreadable state. There were quote blocks that are not quotes, inline code formatting that was not code, and an out-of-context update not marked as an update, edited into the start of the question. I feel that this goes beyond the ordinary level of a merely unclear question, and is reflecting an attachment to over-formatting that borders on vandalism. — halfer, Oct 17 '20 at 10:17

score 2 · Accepted Answer · answered Oct 01 '20 at 13:06

2

The Default Message Time To Live setting does not apply to dead-letter. As per documentation:

There's no automatic cleanup of the DLQ. Messages remain in the DLQ until you explicitly retrieve them from the DLQ and call Complete() on the dead-letter message.

The code above for the function that returns dead-lettered messages to the original queue will complete each dead-lettered message and those will be gone from the dead-letter queue.

answered Oct 01 '20 at 13:06

Sean Feldman

23,443
7
55
80

Thanks for clarification.. could you please help to achieve my goal .. I want to persist that message into dlq. By using above code I'm throwing that message back again to main queue for process and in main function app if still error then it again throw it to dlq – Neo Oct 02 '20 at 15:01
2

To be honest, I don't quite understand what is it that you want to do. DLQ is not a normal queue. It's a system queue. It cannot used to "persist" messages. If a messaged was dead-lettered, goes through the 2nd functions (retry) function, and still ends up in the DLQ, either the message is poisonous or there's a logical issue. – Sean Feldman Oct 03 '20 at 01:19
question updated with Main function code. My logic is until API is up I'm sending message into DLQ and DLQ function app pushing it back to main queue so that i don't lost message. – Neo Oct 03 '20 at 09:15
1

You’ll not lose a message but it might go into infinite loop with the setup you’ve got there. – Sean Feldman Oct 04 '20 at 14:43
but it is losing message after 10 minutes :( yea you are right it may go to infinite loop and stop function app but api downtime is mostly 10-20 minutes – Neo Oct 05 '20 at 05:16
1

Loosing messages after 10 mins? That's not possible. Either message gets processed or somehow completed. A dead-lettered messages should never be lost. If that would happen it would be a broker bug. – Sean Feldman Oct 05 '20 at 05:50
ohh i see got you thanks , do you feel my DLQ function app code need any modification like while pushing message again send it with new message id? – Neo Oct 05 '20 at 05:53
2

I would add to the dead-lettered message a custom property to count how many times it was retried from the dead-lettered queue, a value carried from the original message and increased by one each time the message is sent for re-processing. And at some point rather than retry it, do something else with it (log, store somewhere, etc) to break the loop. – Sean Feldman Oct 05 '20 at 13:19
after some troubleshoot got the root cause -Microsoft.Azure.ServiceBus.ServiceBusCommunicationException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full ErrorCode: NoBufferSpaceAvailable ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was fullat Microsoft.Azure.ServiceBus.ServiceBusConnection.CreateConnectionAsync(TimeSpan timeout) – Neo Oct 08 '20 at 09:54
@Neo, if the exception contained a TrackingID, this is something you could use to request more information about what happened on the broker side. Otherwise, it might be an issue with either the code you're using or the environment. – Sean Feldman Oct 16 '20 at 03:51

Azure service bus deadletter queue message unable to stay in deadletter for longer time

Update

1 Answers1

Linked