0

My question isn't very easy but I believe I will find with Your help the solution for my problem. I have a microservice that reads messages from AWS SQS queue and saves it in Redis.

On AWS I have two queues:

  1. AQueue (standard queue)
  2. DeadLetterQueue

I'd like to:

  • Remove messages from my standard queue (AQueue) and move them to DeadLetterQueue when for example there is 5 times error parsing
  • When for example my Redis is temporarily unavailable, I'd like to NOT to remove currently read messages. In This case these messages should be read over and over again until the Redis will work.

HOW can I do this? On AWS I have set my standard queue (AQueue) to send messages to DeadLetterQueue when the messages will fail 5 times

My listener:

@SqsListener(value = "${amazon.sqs.destination}", deletionPolicy = SqsMessageDeletionPolicy.NEVER)
    public void receive(String requestJSON, Acknowledgment acknowledgment) {
        try (Jedis jedis = jedisPool.getResource()) {
            if (redisPassword != null && !redisPassword.isEmpty()) {
                jedis.auth(redisPassword);
            }
            long key = jedis.incr("Trace:");
            Trace trace = Trace.fromJSON(requestJSON);
            trace.setTechnicalId(Long.toString(key));
            traceRepository.save(trace);
            acknowledgment.acknowledge();
        }catch (IOException e) {
            log.error("Parse error: " + e.getMessage());
            queueMessagingTemplate.convertAndSend(deadLetterQueue, requestJSON);
            acknowledgment.acknowledge();
        } catch(Exception e){
            log.error("Problem with NOSQL database Redis: " + e.getMessage());
        }

Unfortunately, EVEN WHEN I don't call acknowledgment.acknowledge(); my message after 5 attempts is moving to DeadLetterQueue.

Matley
  • 1,953
  • 4
  • 35
  • 73

2 Answers2

0

It sounds like you've got the logic reversed - NOT acking the message will mean that it'll be treated like a failure (and retried, and eventually moved to the dead-letter-queue). ACK the message when you want to mark it as consumed, don't ACK it if you want it to retry/DLQ.

(Note - I'm not familiar with the spring tooling involved here, but I'm assuming straightforward mapping to core SQS concepts)

Krease
  • 15,805
  • 8
  • 54
  • 86
  • Hmmm, ok so what should I do to NOT to ACK my message (the second catch clause)? I thought that with deletionPolicy = SqsMessageDeletionPolicy.NEVER the only one possibility to pull my message is when i call acknowledgment.acknowledge(); Unfortunately my message is gone from my main queue even when I dont call acknowledgment.acknowledge() - but this appears ONLY if I link my main queue with Dead Letter Queue. – Matley Oct 25 '18 at 10:53
0

The logic should be:

  • If Redis is down, do not pull messages from SQS (this will keep them in the queue)
  • If a message is successfully processed, delete the message using the supplied MessageHandle
  • Configure the Queue to move messages to the Dead Letter Queue after 5 processing attempts
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Thank you @John Rotenstein for your answer - but it's all clear for me what you've writter. The question is still the same - how to NOT TO PULL messages from SQS when Redis is down. When Redis is down, the second catch is relevant, but in this catch I don't call acknowledgment.acknowledge(); Unfortunately, despite this the message is moving to the Dead Letter Queue (what's not my goal) – Matley Oct 25 '18 at 10:47
  • The easiest way is probably to have your system (that is pulling message from SQS) stop when it notices that Redis is not responding, then continue polling Redis until it is healthy again. While Redis is not healthy, it should not pull any more messages from SQS. – John Rotenstein Oct 25 '18 at 11:26