10

I am looking for a good way to implement retries with a backoff policy using spring amqp and Rabbit MQ, but the requirement is that the listener should not be blocked (so it is free to process other messages). I see a similar question asked/answered here but it does not include the solution for 'backing off':

RabbitMQ & Spring amqp retry without blocking consumers

The questions I have are:

  1. Does the default spring-retry implementation block threads while retrying? The implementation in github indicates that it does.

  2. If the assumption above is true, is the only way to do this implementing a separate queue for retries (DLQ?), and setting a TTL for each message (assuming we don't want to block threads for the backoff interval).

  3. If we go with the approach above (DLQ or a separate queue), won't we need separate queues for each retry attempt? If we use just 1 queue for retries, the same queue will contain messages with a TTL ranging from minimum retry interval to maximum retry interval, and if the message at the front of the queue has the max TTL, the message behind it will not be picked up even if it has min TTL. This is per the Rabbit MQ TTL documentation here (see Caveats):

  4. Is there another way to implement a non-blocking Backoff Retry mechanism?

Adding some configuration information to help troubleshoot @garyrussel:

Queue Config:

    <rabbit:queue name="regular_requests_queue"/>
    <rabbit:queue name="retry_requests_queue">
        <rabbit:queue-arguments>
            <entry key="x-dead-letter-exchange" value="regular_exchange" />
        </rabbit:queue-arguments>
    </rabbit:queue>

    <rabbit:direct-exchange name="regular_exchange">
        <rabbit:bindings>
            <rabbit:binding queue="regular_requests_queue" key="regular-request-key"/>
        </rabbit:bindings>
    </rabbit:direct-exchange>

    <rabbit:direct-exchange name="retry_exchange">
        <rabbit:bindings>
            <rabbit:binding queue="retry_requests_queue"/>
        </rabbit:bindings>
    </rabbit:direct-exchange>

    <bean id="retryRecoverer" class="com.testretry.RetryRecoverer">
         <constructor-arg ref="retryTemplate"/>
         <constructor-arg value="retry_exchange"/>
    </bean>

    <rabbit:template id="templateWithOneRetry" connection-factory="connectionFactory" exchange="regular_exchange" retry-template="retryTemplate"/>
    <rabbit:template id="retryTemplate" connection-factory="connectionFactory" exchange="retry_exchange"/>

    <bean id="retryTemplate" class="org.springframework.retry.support.RetryTemplate">
        <property name="retryPolicy">
            <bean class="org.springframework.retry.policy.SimpleRetryPolicy">
                <property name="maxAttempts" value="1"/>
            </bean>
        </property>
    </bean>
Pang
  • 9,564
  • 146
  • 81
  • 122
alam86
  • 101
  • 1
  • 5

3 Answers3

0
  1. Yes
  2. thru 4 ...

You could use retry max attempts = 1 with a subclass of RepublishMessageRecoverer and implement additionalHeaders to add, say a retry count header.

You could then republish to a different queue for each attempt.

The recoverer is not really structured to publish to different queues (we should change that), so you might need to write your own recoverer and delegate to one of several RepublishMessageRecoverer.

Consider contributing your solution to the framework.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Thanks for the response @gary! I was going down a similar path, but using a custom 'RejectAndDontRequeueRecoverer', which would then throw 'AmqpRejectAndDontRequeueException'. That should have put the message in the configured DLX, correct? That did not work though. I also tried using a custom 'RepublishMessageRecoverer', which is the better option, but I don't see the message getting 'republished' to the 'retry_exchange'. I do see a message in the log file: 'Republishing failed message to exchange retry_exchange'. – alam86 Sep 18 '15 at 22:26
  • FYI, I also set TTL (expiration) to '60000' (60s) for the message before calling 'super.recover()' in the custom RepublishMessageRecoverer. I do see the custom RetryRecoverer being used (by setting break points), but don't see the message getting published to the retry_requests_queue. See configuration in original message above. – alam86 Sep 18 '15 at 22:48
  • I don't see any configuration for `subscription_exchange`. Shouldn't you be routing to `retry_exchange` ? – Gary Russell Sep 18 '15 at 23:17
  • Sorry - that should have been `regular_exchange`. The idea is that once we set TTL on the message, and it does not get picked up by any listener from the 'retry_requests_queue', after the TTL, it should be dead lettered to the 'regular_exchange' (and get retried). Isn't that how we would implement the backoff? – alam86 Sep 18 '15 at 23:39
  • It wasn't clear which of your situations that configuration represents; the first case where RMQ didn't route to the DLX as expected or when you were republishing and the message wasn't delivered to the queue. Plus, having the wrong exchange name confused things further. What are you using as the routing key for the republish? Your configuration expects the queue name to be used; by default, the recoverer uses `error.`. – Gary Russell Sep 19 '15 at 01:47
  • I am trying to implement the 'Republish' scenario now. Sorry about the confusion with the Exchange names! For the routing key, I left it as the default - so it would be 'error.'. Could you elaborate on what you mean by 'configuration expects the queue name to be used'? In the Republish initialization, I specified 'retry_exchange' and a separate 'retry_template'. The original message now has the template configuration as well. – alam86 Sep 21 '15 at 17:48
  • You don't have any binding set up with that key. You have ``. You need ` – Gary Russell Sep 21 '15 at 18:14
  • Thanks Gary! I had actually tried with the binding too, but it kept failing because I didn't realize I had to delete/recreate the queues. Once I deleted the queues with the old config, (and they got recreated with the new config) the message got routed to the retry queue fine. There may be a problem with this solution though - will describe that in the next post (too long for this one) – alam86 Sep 21 '15 at 21:52
  • Here is the problem I mentioned in my previous post: If the routing key gets changed to 'error.' during republish, when the message expires on the retry_requests_queue, and is dead lettered to the original exchange (regular_exchange), the key will be 'error.', unless I specify a separate key to use for dead_lettered messages, and will not get routed to the original queue. I may have more than one queue (with a different key) in regular_exchange whose requests I'll want to retry using this mechanism (going to the same retry_requests_queue). – alam86 Sep 21 '15 at 22:08
  • So, why not simply set the `errorRoutingKeyPrefix` to `""` (instead of the default `"error."`), and then the original routing key will be used throughout. The javadoc explicitly says `Use an empty string ("") for no prefixing.` – Gary Russell Sep 21 '15 at 22:14
  • Thank you for pointing that out - I missed that in the javadoc! I will try to get a complete solution working and then post it here. Thank you for all your help! – alam86 Sep 22 '15 at 00:07
0

Here is the final solution I ended up implementing. There is 1 queue per 'retry interval', 1 exchange per retry queue. They are all passed to the custom RepublishRecoverer which creates a list of recoverers.

A custom header called 'RetryCount' is added to the Message, and depending on the value of 'RetryCount', the message is published to the right exchange/queue with a different 'expiration'. Each retry queue is setup with a DLX, which is set to the 'regular_exchange' (i.e. requests go to the regular queue).

<rabbit:template id="genericTemplateWithRetry" connection-factory="connectionFactory" exchange="regular_exchange" retry-template="retryTemplate"/>

<!-- Create as many templates as retryAttempts (1st arg) in customRetryTemplate-->
<rabbit:template id="genericRetryTemplate1" connection-factory="consumerConnFactory" exchange="retry_exchange_1"/>
<rabbit:template id="genericRetryTemplate2" connection-factory="consumerConnFactory" exchange="retry_exchange_2"/>
<rabbit:template id="genericRetryTemplate3" connection-factory="consumerConnFactory" exchange="retry_exchange_3"/>
<rabbit:template id="genericRetryTemplate4" connection-factory="consumerConnFactory" exchange="retry_exchange_4"/>
<rabbit:template id="genericRetryTemplate5" connection-factory="consumerConnFactory" exchange="retry_exchange_5"/>

<rabbit:queue name="regular_requests_queue"/>

<!-- Create as many queues as retryAttempts (1st arg) in customRetryTemplate -->
<rabbit:queue name="retry_requests_queue_1">
    <rabbit:queue-arguments>
        <entry key="x-dead-letter-exchange" value="regular_exchange" />
    </rabbit:queue-arguments>
</rabbit:queue>
<rabbit:queue name="retry_requests_queue_2">
    <rabbit:queue-arguments>
        <entry key="x-dead-letter-exchange" value="regular_exchange" />
    </rabbit:queue-arguments>
</rabbit:queue>
<rabbit:queue name="retry_requests_queue_3">
    <rabbit:queue-arguments>
        <entry key="x-dead-letter-exchange" value="regular_exchange" />
    </rabbit:queue-arguments>
</rabbit:queue>
<rabbit:queue name="retry_requests_queue_4">
    <rabbit:queue-arguments>
        <entry key="x-dead-letter-exchange" value="regular_exchange" />
    </rabbit:queue-arguments>
</rabbit:queue>
<rabbit:queue name="retry_requests_queue_5">
    <rabbit:queue-arguments>
        <entry key="x-dead-letter-exchange" value="regular_exchange" />
    </rabbit:queue-arguments>
</rabbit:queue>

<rabbit:direct-exchange name="regular_exchange">
    <rabbit:bindings>
        <rabbit:binding queue="regular_requests_queue" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

<!-- Create as many exchanges as retryAttempts (1st arg) in customRetryTemplate -->
<rabbit:direct-exchange name="retry_exchange_1">
    <rabbit:bindings>
        <rabbit:binding queue="retry_requests_queue_1" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

<rabbit:direct-exchange name="retry_exchange_2">
    <rabbit:bindings>
        <rabbit:binding queue="retry_requests_queue_2" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

<rabbit:direct-exchange name="retry_exchange_3">
    <rabbit:bindings>
        <rabbit:binding queue="retry_requests_queue_3" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

<rabbit:direct-exchange name="retry_exchange_4">
    <rabbit:bindings>
        <rabbit:binding queue="retry_requests_queue_4" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>

<rabbit:direct-exchange name="retry_exchange_5">
    <rabbit:bindings>
        <rabbit:binding queue="retry_requests_queue_5" key="v1-regular-request"/>
    </rabbit:bindings>
</rabbit:direct-exchange>


<!-- retry config begin -->
<!-- Pass in all templates and exchanges created as list/array arguments below -->
<bean id="customRetryRecoverer" class="com.test.listeners.CustomRetryRecoverer">
    <!-- Pass in list of templates -->
     <constructor-arg>
        <list>
            <ref bean="genericRetryTemplate1"/>
            <ref bean="genericRetryTemplate2"/>
            <ref bean="genericRetryTemplate3"/>
            <ref bean="genericRetryTemplate4"/>
            <ref bean="genericRetryTemplate5"/>
        </list>
     </constructor-arg>
     <!-- Pass in array of exchanges -->
     <constructor-arg value="retry_exchange_1,retry_exchange_2,retry_exchange_3,retry_exchange_4,retry_exchange_5"/>
     <constructor-arg ref="customRetryTemplate"/>
</bean>

<bean id="retryInterceptor"
      class="org.springframework.amqp.rabbit.config.StatefulRetryOperationsInterceptorFactoryBean">
    <property name="messageRecoverer" ref="customRetryRecoverer"/>
    <property name="retryOperations" ref="retryTemplate"/>
    <property name="messageKeyGenerator" ref="msgKeyGenerator"/>
</bean>
    
<bean id="retryTemplate" class="org.springframework.retry.support.RetryTemplate">
    <property name="retryPolicy">
        <bean class="org.springframework.retry.policy.SimpleRetryPolicy">
            <!--  Set to 1 - just for the initial attempt -->
            <property name="maxAttempts" value="1"/>
        </bean>
    </property>
</bean>

 <bean id="customRetryTemplate" class="com.test.retry.CustomRetryTemplate">
    <constructor-arg value="5"/> <!-- max attempts -->
    <constructor-arg value="3000"/> <!-- Initial interval -->
    <constructor-arg value="5"/> <!-- multiplier for backoff -->
</bean>

<!-- retry config end -->

Here is the code for the CustomRetryRecoverer:

public class CustomRetryRecoverer extends
        RepublishMessageRecoverer {

    private static final String RETRY_COUNT_HEADER_NAME = "RetryCount";
    private List<RepublishMessageRecoverer> retryExecutors = new ArrayList<RepublishMessageRecoverer>();
    private TriggersRetryTemplate retryTemplate;
    
    public TriggersRetryRecoverer(AmqpTemplate[] retryTemplates, String[] exchangeNames, TriggersRetryTemplate retryTemplate) {
        super(retryTemplates[0], exchangeNames[0]);
        this.retryTemplate = retryTemplate;

        //Get lower of the two array sizes
        int executorCount = (exchangeNames.length < retryTemplates.length) ? exchangeNames.length : retryTemplates.length;
        for(int i=0; i<executorCount; i++) {
            createRetryExecutor(retryTemplates[i], exchangeNames[i]);
        }
        //If not enough exchanges/templates provided, reuse the last exchange/template for the remaining retry recoverers
        if(retryTemplate.getMaxRetryCount() > executorCount) {
            for(int i=executorCount; i<retryTemplate.getMaxRetryCount(); i++) {
                createRetryExecutor(retryTemplates[executorCount-1], exchangeNames[executorCount-1]);
            }
        }
    }

    @Override
    public void recover(Message message, Throwable cause) {
        
        if(getRetryCount(message) < retryTemplate.getMaxRetryCount()) {
            incrementRetryCount(message);
            
            //Set the expiration of the retry message
            message.getMessageProperties().setExpiration(String.valueOf(retryTemplate.getNextRetryInterval(getRetryCount(message)).longValue()));
            
            RepublishMessageRecoverer retryRecoverer = null;
            if(getRetryCount(message) != null && getRetryCount(message) > 0) {
                retryRecoverer = retryExecutors.get(getRetryCount(message)-1);
            } else {
                retryRecoverer = retryExecutors.get(0);
            }
            retryRecoverer.recover(message, cause);
        } else {
            //Retries exchausted - do nothing
        }
    }

    private void createRetryExecutor(AmqpTemplate template, String exchangeName) {
        RepublishMessageRecoverer retryRecoverer = new RepublishMessageRecoverer(template, exchangeName);
        retryRecoverer.errorRoutingKeyPrefix(""); //Set KeyPrefix to "" so original key is reused during retries
        retryExecutors.add(retryRecoverer);
    }   

    private Integer getRetryCount(Message msg) {
        Integer retryCount;
        if(msg.getMessageProperties().getHeaders().get(RETRY_COUNT_HEADER_NAME) == null) {
            retryCount = 1;
        } else {
            retryCount =  (Integer) msg.getMessageProperties().getHeaders().get(RETRY_COUNT_HEADER_NAME);
        }
        
        return retryCount;
    }

    private void incrementRetryCount(Message msg) {
        Integer retryCount;
        if(msg.getMessageProperties().getHeaders().get(RETRY_COUNT_HEADER_NAME) == null) {
            retryCount = 1;
        } else {
            retryCount =  (Integer) msg.getMessageProperties().getHeaders().get(RETRY_COUNT_HEADER_NAME)+1;
        }
        msg.getMessageProperties().getHeaders().put(RETRY_COUNT_HEADER_NAME, retryCount);
    }

}

The code for 'CustomRetryTemplate' is not posted here, but it contains simple variables for maxRetryCount, initialInterval and multiplier.

Pang
  • 9,564
  • 146
  • 81
  • 122
alam86
  • 101
  • 1
  • 5
0

Did you look at rabbitmq delayer plugin that delays the messages at the exchange instead of queue? As per documentation, messages sent to the delayer exchange seems to be persistent at the exchange level.

Using the custom retry count message header & the delayer exchange, we can achieve the non-blocking behaviour without the ugliness of these intermediate queue, dlx & template combination

https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/

Ashok Koyi
  • 5,327
  • 8
  • 41
  • 50
  • This does look like a viable option @kalinga. I have not tried it yet - the version we have is 3.3.5. It also sounds like an 'experimental' implementation right now. Once it gets more mature, it is probably the preferred option for this. The solution posted above does work for us with the version of rabbitmq we have. – alam86 Aug 16 '16 at 17:48