RabbitMQ - Apache Camel Reading Messages what to do with failed messages

Question

I have the following PHP application. That publishes a user signUp to a message queue. The Java Application reads from that queue and imports it. Hopefully the diagram below will discribe it. I am only working with the Java side of things. The json messages exists on the queue already.

Route (Java Consuming Side).

@Component
public class SignUpRouting {

  errorHandler(deadLetterChannel("rabbitmq://signUpDeadLetter.exchange?username=etc..").useOriginalMessage());

  from("rabbitmq://phpSignUp.exchange?username=etc....")
            .routeId("signUpRoute")
            .processRef("signUpProcessor")
            .end();
  //....

The processor..

@Component
public class SignupProcessor implements Processor {

    private ObjectMapper mapper = new ObjectMapper();

    @Override
    public void process(Exchange exchange) throws Exception {

        String json = exchange.getIn().getBody(String.class);
        SignUpDto dto = mapper.readValue(json, SignUpDto.class);

        SignUp signUp = new SignUp();
        signUp.setWhatever(dto.getWhatever());
        //etc....

        // save record
        signUpDao.save(signUp);
    }
}

My question is this.. What should I do I do when the Processor fails to import the message.

Lets say for example there was a DAO exception. A data field may have been toolong or the import was in the incorrect format. I dont want to lose the message. I would like to see the error and retry the import. But I would not want to keep retrying the message every 30 seconds.

I am thinking that I would need to create another queue.. A dead letter queue and have that indefinately retry the message every 6 hours?.. I would then view the logs see the error and upload a fix and the message would be reprocessed?

How would I implement that? Or am I on the wrong track?

EDIT I have tried setting deadLetterExchange to see if would get things on the right direction... However it errors and says queue cannot be non null

 rabbitmq://phpSignUp.exchange?username=etc...&deadLetterExchange=signUpDeadLetter.exchange

if you are using another queue why don't you store the exact failed message along with the stack trace of the exception and then process data from that queue — Lalit Mehra, Oct 19 '15 at 10:04
If you have the luxury of a support team, I would send the message to another queue or write to a database table and then send an email to alert support staff. Create another interface that allows the support staff to modify the text of the message and reinject it into the signup processor. Whatever you do is going to require manual intervention. Design accordingly. There should be good validation on the PHP app such that his is a rare event. — Sammy, Oct 20 '15 at 09:49

sanigo · Accepted Answer · 2015-11-01T15:16:41.617

2

Here is a example to use dead letter headers:

        <from uri="rabbitmq://localhost/youexchange?queue=yourq1&amp;
            exchangeType=topic&amp;
            routingKey=user.reg.*&amp;
            deadLetterExchange=dead.msgs&amp;
            deadLetterExchangeType=topic&amp;
            deadLetterRoutingKey=dead.letters&amp;
            deadLetterQueue=dead.letters&amp;
            autoAck=false&amp;
            autoDelete=false"/>

          <!--We can use onException to make camel to retry, and after that, dead letter queue are the fallback-->
        <onException useOriginalMessage="true">
            <exception>java.lang.Exception</exception>
            <redeliveryPolicy asyncDelayedRedelivery="true" maximumRedeliveries="3" redeliveryDelay="5000"/>
        </onException>

We need to turn off autoAck and set deadLetterQueue, then if there is an Exception thrown, the message will be in dead letters queue. To use onException, we can control the retrying before camel dropping the message to dead letter queue.

edited Nov 01 '15 at 15:16

answered Oct 23 '15 at 13:29

sanigo

625
4
14

Couple of questions. Why do we need to turn of autoAck? and what is deadLetterExchangeType=topic? – Robbo_UK Oct 31 '15 at 15:14
If the autoAck is on, camel will send basic.ack while they receive messages. There will be no possible for camel to send a basic.refuse or basic.nack upon excepions, so the deadletter attributes will be useless. – sanigo Nov 01 '15 at 14:17
And the deadLetterExchangeXXX attributes are used to route dead letters to the specified exchange and queue. In the demo settings, i used topic, but you can use other exchange types. – sanigo Nov 01 '15 at 14:19

score 1 · Answer 2 · edited Jun 20 '20 at 09:12

1

You could use onException to catch Exceptions, if there is an Exception, the message will be route to the dead letter exchange, here is the example in Spring DSL:

<onException useOriginalMessage="true">
            <exception>java.sql.SQLException</exception>
            <redeliveryPolicy asyncDelayedRedelivery="true" maximumRedeliveries="1" redeliveryDelay="1000"/>

            <inOnly uri="rabbitmq://localhost/dead.msgs?exchangeType=fanout&amp;
                    autoDelete=false&amp;
                    bridgeEndpoint=true"/>
</onException>

edited Jun 20 '20 at 09:12

Community

1
1

answered Oct 22 '15 at 04:13

sanigo

625
4
14

This is a good answer and a viable solution +1. However it feels to me like its not using the rabbitmq specific dead letter headers and config. As seen http://camel.apache.org/rabbitmq.html – Robbo_UK Oct 22 '15 at 09:25
why do we need to set bridgeEndpoint=true. what is that for? – Robbo_UK Oct 31 '15 at 15:14
If you do not add the attribute, the target queue will not receive any messages. – sanigo Nov 01 '15 at 14:20

RabbitMQ - Apache Camel Reading Messages what to do with failed messages

2 Answers2