I have a problem with a webjob destroying messages from service bus topic if an exception occurs. I want the message to stay on the topic (for the subscription) until it processes successfully. Even if that means messages are behind it are blocked until the first message succeeds.
We have a web job that processes messages on a Service Bus Topic using a trigger, example below:
[Singleton]
public void ProcessQueueMessage([ServiceBusTrigger("%TopicNameEvent%", "%SubscriptionNameEvent%")] BrokeredMessage message, TextWriter log)
{
lock (MsgLock)
{
log.WriteLine($"{DateTime.UtcNow} - Event processing started for Message Id {message.MessageId}.");
var proc = new MessageProcessor();
proc.ProcessBrokeredMessage(message);
log.WriteLine($"{DateTime.UtcNow} - Event processing finished for Message Id {message.MessageId}.");
}
}
In the ProcessBrokeredMessage() call, I will fabricate an exception being thrown like this:
public void ProcessBrokeredMessage(BrokeredMessage message)
{
try
{
throw new Exception("blah blah blah");
}
catch (Exception ex)
{
_log.Error(ex); // Log what caused exception
throw; // Rethrow in the hopes that the message will stick and not be abandoned ....
}
}
What I find happens is that when the exception occurs, the message is abandoned, and then the subsequent message is triggered. It did not retry the message that threw the exception? My understanding of the SDK is that it should only complete the message processing on success and that an exception would keep the message alive for the subscription?
This is how we create the Service Bus client configuration:
_subClient = SubscriptionClient.CreateFromConnectionString(_servicesBusConnectionString, _feedName, _subscriptionName);
_subClient.RetryPolicy = RetryPolicy.Default;
var config = new JobHostConfiguration
{
JobActivator = new NinjectActivator(_ninjectKernel),
NameResolver = new TopicNameResolver()
};
var sbConfig = new ServiceBusConfiguration {ConnectionString = _servicesBusConnectionString};
sbConfig.MessageOptions.MaxConcurrentCalls = 1; // We only want to process one message at a time, in order
config.UseServiceBus(sbConfig);
// block the host process
_host = new JobHost(config);
_host.RunAndBlock();
I did try some code about setting the Dequeue count (example: Queues = {MaxDequeueCount = 100 } // try 100 times) But that doesn't work either.
UPDATE:
DeadLetterMessageCount remains the same before and after processing the message that raises an exception. Before processing is 14. After processing remains at 14:
UPDATE #2:
Okay, so I decided to write the Message IDs to a text file when the exception occurs. Looks like it does try the message again when an exception occurs. But no guarantees as to when it does ... Bolded IDs below that are the same, to show when they tried again
1932248 1932324 1932259 1932233 1932308 1932249 1932250 1932240 1932333 1932226 1932231 1932247 1932229 1932234 1932334 1932232 1932220 1932259 1932333 1932229 1932234 1932220 1932226 1932247 1932248 1932250 1932240 1932233 1932232 1932249 1932324 1932231 1932334 1932259 1932334 1932229 1932333 1932226 1932232 1932247 1932248 1932240 1932249 1932324 1932234 1932250 1932424 1932233 1932231 1932333 1932334
UPDATE #3: Ok, I can confirm it will eventually retry the message 10 times before moving to the Dead Letter Queue.
I just need to figure out now how to make the web job stop processing everything when an exception occurs, until the same message can succeed (most likely wait for a human to fix something, redeploy new code, fix database issue, etc).
