Rebus Retry policy

Question

I have questions on Rebus retry policy below:

Configure.With(...)
    .Options(b => b.SimpleRetryStrategy(maxDeliveryAttempts: 2))
    .(...)

https://github.com/rebus-org/Rebus/wiki/Automatic-retries-and-error-handling#customizing-retries

1 Can it be used for both Publiser (enqueue messages) and Subscriber (dequeue messages)?

2 I have a subscriber that is unable to dequeue the message. Thus the message is sent to error queue.

Below is error for when putting the message to error queue. But I cannot see the loggings for the retry.

[ERR] Rebus.Retry.PoisonQueues.PoisonQueueErrorHandler (Thread #9): Moving messa
ge with ID "<unknown>" to error queue "poison"
Rebus.Exceptions.RebusApplicationException: Received message with empty or absen
t 'rbs2-msg-id' header! All messages must be supplied with an ID . If no ID is p
resent, the message cannot be tracked between delivery attempts, and other stuff
 would also be much harder to do - therefore, it is a requirement that messages
be supplied with an ID.

Is it possible to define and store custom logging for each retry, not within IErrorHandler?

3 How long does each retry wait in between be default?

4 Is it possible to define custom wait time for each retry (not within IErrorHandler)? If so, is Polly supported for this scanario? like below:

Random jitterer = new Random(); 
Policy
  .Handle<HttpResponseException>() // etc
  .WaitAndRetry(5,    // exponential back-off plus some jitter
      retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))  
                    + TimeSpan.FromMilliseconds(jitterer.Next(0, 100)) 
  );

https://learn.microsoft.com/en-us/dotnet/standard/microservices-architecture/implement-resilient-applications/implement-http-call-retries-exponential-backoff-polly

Update

How can I test the retry policy?

Below is what I tried based on the code below:

public class StringMessageHandler : IHandleMessages<String>
{   
    public async Task Handle(String message) 
    {
          //retry logic with Polly here
    }   
}

I sent an invalid message of string type to the string topic, however, the Handle(String message) is not invoked at all.

What do you mean by `(...) I sent an invalid message of string type to the string topic (...)`? Did you publish it? Did you remember to subscribe to it? — mookid8000, May 21 '19 at 19:30
I sent an invalid message using Azure Service Bus Explorer. The message only has a body, no header. I didn't publish it via Rebus. This is only to test the retry logic, but the Handle() is not called. However, the invalid message is moved to error queue by Rebus. Is it possible to hook into the retry logic so that logging and Polly policy can be performed? My understanding of Handle method might be different from Rebus. — Pingpong, May 22 '19 at 00:56
Your message gets immediately moved to the dead-letter queue because it doesn't have a `rbs2-message-id` header – that header is the bare minimum required for Rebus to even try to handle a message, as it's necessary for the retry mechanism to work. — mookid8000, May 22 '19 at 05:58
Sorry for not making it clearer. My main question is how to test the custom retry logic via Polly within StringMessageHandler.Handle(String message) method? — Pingpong, May 22 '19 at 21:48
You class is just a class that implements an interface – it doesn't really contain any Rebus-related logic, so I suggest you just call the `Handle` method and verify that your Polly policy works :) — mookid8000, May 23 '19 at 06:59
At what point the `Handle` method can be called so the retry logic within it can be tested? Under what circumtances, the retry logic within `StringMessageHandler.Handle(String message)` can be tested? — Pingpong, May 23 '19 at 10:28
``` var handler = new StringMessageHandler( (..) dependencies here); await handler.Handle("I wonder if it works"); // assert behavior was correct ``` — mookid8000, May 23 '19 at 11:35
Before calling `Handle` method, Rebus performs retry if necessary, then calls the `Handle` method to process method. My question is that how Rebus tests that it is ok to call `Handle` method? That is, the logic of determining various things before invoking `Handle` method. For example, does it check if the queue is connected? message of correct type exist? What happen if there are error within `Handle` method. For example, exception throws within it, — Pingpong, May 23 '19 at 13:58
When Rebus receives a message, it will try to handle it. This means that the message will have its headers inspected, it will be deserialized, and it will be dispatched to your handlers – all that happens in "the incoming messages pipeline". If some of that fails, Rebus will ROLL BACK the queue transaction, but it will remember its message ID, so that it can count how many times the message has failed. That's how retry works with Rebus — mookid8000, May 24 '19 at 06:29
Your questions are very detailed, so it's hard to answer everything in this little box. I think you should clone the source code and read from here: [The place where an incoming message gets handled](https://github.com/rebus-org/Rebus/blob/master/Rebus/Workers/ThreadPoolBased/ThreadPoolWorker.cs#L102-L123) – then you can step into the methods and see what they do. — mookid8000, May 24 '19 at 06:31

score 2 · Answer 1 · answered May 16 '19 at 11:59

Rebus' retry mechanism is only relevant when receiving messages. It works by creating a "queue transaction"(*), and then if message handling fails, the message gets rolled back to the queue.

Pretty much immediately thereafter, the message will again be received and attempted to be handled. This means that there's no delay between delivery attempts.

For each failed delivery, a counter gets increased for that message's ID. That's why the message ID is necessary for Rebus to work, which also explains why your message with out an ID gets immediately moved to the dead-letter queue.

Because of the disconnected nature of the delivery attempts (only a counter per message ID is stored), there's no good place to hook in a retry library like Polly.

If you want to Pollify your message handling, I suggest you carry out individual operations with Polly policies – that way, you can easily have different policies for dealing with failing web requests, failing file transfers on network drives, etc. I do that a lot myself.

To avoid not being able to properly shut down your bus instance if it's in the process of a very long Polly retry, you can pass Rebus' internal CancellationToken to your Polly executions like this:

public class PollyMessageHandler : IHandleMessages<SomeMessage>
{
    static readonly IAsyncPolicy RetryPolicy = Policy
        .Handle<Exception>()
        .WaitAndRetryForeverAsync(_ => TimeSpan.FromSeconds(10));

    readonly IMessageContext _messageContext;

    public PollyMessageHandler(IMessageContext messageContext)
    {
        _messageContext = messageContext;
    }

    public async Task Handle(SomeMessage message) 
    {
        var cancellationToken = _messageContext.GetCancellationToken();

        await RetryPolicy.ExecuteAsync(DoStuffThatCanFail, cancellationToken);
    }

    async Task DoStuffThatCanFail(CancellationToken cancellationToken)
    {
        // do your risky stuff in here
    }
}

(*) The actual type of transaction depends on what is supported by the transport.

With MSMQ, it's a MessageQueueTransaction object that has Commit() and Rollback() methods on it.

With RabbitMQ, Azure Service Bus, and others, it's a lease-based protocol, where a message becomes invisible for some time, and then if the message is ACKed within that time, then the message is deleted. Otherwise – either if the message or NACKed, or if the lease expires – the message pops up again, and can again be received by other consumers.

With the SQL transports, it's just a database transaction.

If we want to use Polly like PollyMessageHandler, is it correct that maxDeliveryAttempts should be set to 1? So the retry policy from Polly will be used instead. Plus, logging can be defined and customised within Handle() method. — Pingpong, May 16 '19 at 15:51
IMessageContext is injected automitically. Is dependency injection supported for Rebus's types only? That is, custom type requires 3rd party IoC, e.g. AutoFac? — Pingpong, May 16 '19 at 19:08
If you're using one of the IoC libraries for Rebus (e.g. Rebus.Autofac, Rebus.Castle.Windsor, etc) then `IBus`, `IMessageContext`, and `ISyncBus` are automatically injected. If you're using the built-in handler activator, then you can inject your own dependencies in the factory method when you register your handler, e.g. like this: `activator.Register((bus, context) => new MyHandler(...))` — mookid8000, May 16 '19 at 20:23
Hi, Please see my update on my OP regarding the retry policy handler. — Pingpong, May 21 '19 at 11:47
> With the SQL transports, it's just a database transaction. @mookid8000 What happened with message if my handler throws (my database changes are rolledback) and message should be moved to the error queue. I assume that is done not in the current transaction because current will be rolled back, right? — dariol, Sep 02 '20 at 11:54

Rebus Retry policy

1 Answers1