Azure Service Bus BrokeredMessage has already been consumed?

Question

I'm having an intermittent problem with the Azure Service bus. Sporadically, placing a message on the bus causes the following exception:

TYPE: InvalidOperationException

MESSAGE: The operation cannot be performed because the brokered message '723eab13dab34351a78bb687d0923b89' has already been consumed. Please use a new BrokeredMessage instance for the operation.

STACKTRACE

at Microsoft.ServiceBus.Messaging.MessagingUtilities.ValidateAndSetConsumedMessages(IEnumerable`1 messages)
at Microsoft.ServiceBus.Messaging.MessageSender.Send(TrackingContext trackingContext, IEnumerable`1 messages, TimeSpan timeout)
at Microsoft.Practices.TransientFaultHandling.RetryPolicy.<>c__DisplayClass1.<ExecuteAction>b__0()
at Microsoft.Practices.TransientFaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
at IQ.IR.Core.ServiceBus.AzureBus`1.Enqueue(T message) in c:\BuildAgent\work\cc0c51104c02a4e9\IQ.IR.Core\ServiceBus\AzureBus.cs:line 69
...Rest of stacktrace snipped as it's within my app

The offending code from AzureBus is:

public void Enqueue(T message)
{
    using (var brokeredMessage = new BrokeredMessage(message) { Label = message.GetType().FullName, TimeToLive = _timeToLive })
    {
        _retryPolicy.ExecuteAction(() => _sender.Send(brokeredMessage));
    }
}

Where T message being passed in is

[Serializable]
public class ValidationMessage
{
    public string ValidationToken { get; set;}
}

And _retryPolicy is a

RetryPolicy<ServiceBusTransientErrorDetectionStrategy>

_timeToLive is a 12 hour timespan

Any ideas?

score 6 · Answer 1 · answered Sep 07 '12 at 16:22

6

.... and to pile on to Abhishek's answer: Right now you need to construct a new BrokeredMessage for each retry, so your retry policy scope needs to be one level further up. Mind that if you put in a stream, we will use that stream as-is inside the brokered message and pull straight fro it onto the wire, so you will need to make copies of the stream ahead of time for a retry loop.

answered Sep 07 '12 at 16:22

Clemens Vasters

2,666
16
28

I've been looking at sample code everywhere, especially in the WindowsAzure.com website, and they are extremely lacking in these kinds of detail. – Thiago Silva Mar 08 '13 at 17:09
argh....hit enter on the comment....ok, more: Is there a sample app that deals with this scenario, using Streams for the message body, and builds a reliable receive and processing of brokered messages? I've found a page from the WACAT, but it's dated from 2011, and has some old API references. – Thiago Silva Mar 08 '13 at 17:11
Particularly, if you call GetBody, but then your processing action fails, and you have to abandon the message. The stream body is left in a bad state since it's been consumed already, so when the loop hits the message again, it blows up – Thiago Silva Mar 08 '13 at 17:12

score 5 · Accepted Answer · answered Sep 07 '12 at 06:13

5

The error indicates that the message was "already sent" but an error occurred in the process. Unfortunately there is no easy way to know this by inspecting the message and the message cannot be reused again as it is considered consumed. We are working on some improvements such as allowing you to query the state of such a message and throwing a MessagingException instead of InvalidOperation. Finally the ability to clone a message will help to make recovery easier from such failures.

answered Sep 07 '12 at 06:13

Abhishek Lal

3,233
1
17
16

Abhishek Lal - can I assume that message will be delivered or should I retry as @Clemens says below with new BrokeredMessage ? Note: I must avoid case that message will be delivered twice Thank you – Jack Juiceson Dec 16 '12 at 11:21
If you want to guarantee "exactly once" delivery the enable de-duplication on the queue (you will need to provide the messageId property but then we will filter out duplicates for the time window specified) http://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.queuedescription.requiresduplicatedetection.aspx and http://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.queuedescription.duplicatedetectionhistorytimewindow.aspx – Abhishek Lal Feb 28 '13 at 05:51
The legend says that even in 2021, microsoft is still working on fixing this issue... – Cristian E. Aug 25 '21 at 09:15

score 2 · Answer 3 · answered Aug 23 '18 at 13:20

2

I recently ran into this and found another cause: if you are constructing a BrokeredMessage and set a break point prior to sending it with your MessageSender, and inspect the properties on the Message, it will attempt to access the Queue and throw this exception. Simply sending the message first without inspecting its properties in the IDE will not cause this problem.

answered Aug 23 '18 at 13:20

Nathan

31
5

2

This was the case for me! – Utsav Dawn Jun 12 '19 at 20:28

score 1 · Answer 4 · answered Sep 10 '12 at 19:54

1

Is it possible that the message could have been sent, even though there was an error? Should the message be idempotent before a retry is attempted in this case?

answered Sep 10 '12 at 19:54

woaksie

89
1
3
13

Yes, it is possible that a message is sent and there is an error on the acknowledgement side of things, the way to guarantee at-most-once delivery will be to setup De-duplication on the Queue/Subscription using the MessageId property to uniquely identify the messages and the DuplicateDetectionHistoryTimeWindow value – Abhishek Lal May 21 '13 at 19:42

Azure Service Bus BrokeredMessage has already been consumed?

4 Answers4