Process messages from AWS SQS using multiple hosts

Question

I have a application involving AWS SNS, SQS and needs to be run on multiple hosts. The exact problem description is:

Whenever there is an event, a message containing ID is published to a SNS topic which has a SQS queue subscribed to it. Now I get messages in the queue. Now, I want multiple hosts to read messages from the queue (no two hosts should read the same message) and write the message to a common file in Amazon S3. Issues like 'what if the host reading the message fails' and 'not reading the same message twice in the same host or different hosts ' should be considered.

Can anyone suggest some method or some references through which I can go through to achieve this task?

what you are asking for, is exactly what SQS does. You need to be more specific about the problem you are having. — E.J. Brennan, May 06 '16 at 16:15

Frederick Cheung · Answer 1 · 2016-05-06T11:26:47.037

2

It sounds like what you want is largely how SQS behaves by default. When a message is read by one of your hosts, it becomes invisible to other people accessing the queue, up to the message visibility timeout. You can make api calls to extend that timeout (i.e. a sort of heartbeat).

You can also configure a dead letter queue. With this, after a message has been received a certain number of times it is moved to a separate queue, either for inspection or processing in some other way.

This is documented here

edited May 06 '16 at 11:26

answered May 06 '16 at 10:55

Frederick Cheung

83,189
8
152
174

Can you please provide some reference. I was looking at the AWS documentation but did not find one !!! – rightCoder May 06 '16 at 11:23

mootmoot · Answer 2 · 2016-05-06T14:23:39.293

The solution is scattered around. You can read through SQS Dead Letter queue setup and reference to my example. you can do exactly the same things using AWS SQS console without coding.

import boto3
sqs = boto3.client("sqs")
# I want to "lock" my queue for 5 minutes to allow my process have time to 
# complete the task and delete the message afterwards.
response = sqs.create_queue(
    QueueName="foo",
    Attributes= {
        "VisibilityTimeout" : "300"
    }
)
# create a queue to store the "dead letter message"
dlq_response = sqs.create_queue(
    QueueName="dlq-foo",
    Attributes= {
        "VisibilityTimeout" : "300"
    }
)
queue_url = response["QueueUrl"]

# Attach RedrivePolicy to drive message to dead letter queue 
# I want to make sure the message only read 1 time. Assume the program crash
# if it is not deleted.
# deadLetterTargetArn : You must specify the queue exact region name, 
# exact Account name(replace 1234567890) and your dead letter queue name dlq-foo
sqs.set_queue_attributes(
    QueueUrl = queue_url,
    Attributes = {
        "RedrivePolicy" : """{
                "maxReceiveCount" : "1" ,
                "deadLetterTargetArn" : "arn:aws:sqs:<region-name>:1234567890:dlq-foo"
            }"""
        }
    )

Note : The RedrivePolicy only access literal strings, NOT dictionary. However,as the doucumentation point out, you need to put "dictionary like" value in there and format it as string. You may use a str(dict()) to convert dict to string , I use python triple-quotes quotes for reading clarity .

Process messages from AWS SQS using multiple hosts

2 Answers2