SNS publishes to the SQS queue but the SNS subscription is not consuming the messages published to the SQS queue

Question

Do I have to consume the message separately with SQS? Publishing my message with SNS and consuming with SQS feels weird to me.

This is the pubsub implementation

export const awsSNSPlugin = fp(async server => {
    const snsService = new SNS({ apiVersion: '2010-03-31' });
    const publish: AWSPubSub['publish'] = async params => {
        try {
            const response = await snsService.publish(params).promise();

            server.log.info(`Message ${params.Message} sent to the topic ${params.TopicArn}`);
            server.log.info(response);
        } catch (err: any) {
            server.log.error(err, err.stack);
        }
    };

    const subscribe: AWSPubSub['subscribe'] = async (params, callback) => {
        try {
            const response = await snsService.subscribe(params).promise();
            server.log.info(`this is subscribe response: ${JSON.stringify(response)}`);
        } catch (err: any) {
            server.log.error(err, err.stack);
        }
    };

    server.decorate('pubSub', {
        publish,
        subscribe,
    });
});

I will call the method publish to publish a message like "test message", which will output "Message test message sent to the topic testTopic" on successful publish. I expect the subscriber to then respond with "this is subscribe response: test message" , but it doesn't. It only fires when the subscriber is first initialized on server start with this message:

this is subscribe response: {"ResponseMetadata":{"RequestId":"e4e6b056-f388-5f3d-8a8b-eb884551eaa1"},
"SubscriptionArn":"arn:aws:sns:us-east-1:111111111111111:profileCreated:12hello34-3hi4-1234-hello-12b12bff1212"}

I can see that SNS topic is publishing to the SQS because the number of message goes up in the SQS management console. However, the SNS subscription function is not consuming the messages. It outputs the console log (the callback function) when the server is booted up, but it does not respond to any of the published messages.

Huh? Your title says "SNS publishes to the SQS queue", so why is it also asking about "consuming the messages published in the SQS queue"? Isn't that backwards? Or do you have multiple SNS Topics? — John Rotenstein, Nov 04 '22 at 13:24
I see from the management console that SNS publishes to the SQS but my SNS subscribe function in the fastify server doesnt receive the published messages and my suspicion is that I need to consume the messages through SQS receive message not through SNS subscribe method. — Mike Yim, Nov 04 '22 at 17:15
Your question is really confusing, and your attempt at clarification in the comments is just as confusing. Why do you have SNS configured to publish to SQS, if you want to receive the message through an "SNS Subscribe Function"? Why are you even using SQS? — Mark B, Nov 04 '22 at 17:20
I expect the console.log `this is subscribe response: {$JSON.stringify(response)}` to fire every time I publish a message, which fires, for example, "Message profile-created sent to the topic profileCreated". When I publish a message, I can see from my AWS management console that the message is going into the SQS queue, so publish works. But I am not getting back "this is subscribe response: profile created" console log. My suspicion is that SNS subscribe function doesn't really consume the messages being pushed to SQS. I'm using SNS and SQS to replace RabbitMQ eventbus I previously had. — Mike Yim, Nov 04 '22 at 19:11
If you take a look at this repo: https://github.com/markcallen/snssqs you can see that this person publishes to SNS topic using the SNS object from aws-sdk and consumes the messages through SQS object from aws-sdk. I'm just asking if there is a way to use just the SNS object to handle both pub and sub and not rely on separate SQS object to consume the messages that are being published. Hope that clarifies it. — Mike Yim, Nov 04 '22 at 19:23

score 0 · Accepted Answer · answered Nov 04 '22 at 21:20

I think that you are asking how the Subscribe functionality works with an Amazon SNS topic.

Think of an Amazon SNS topic as a mailing list. Any message sent to the Topic will be forwarded to everything that has subscribed to the topic. If there are 10 subscribers, then then sending one message to the Topic will cause the message to be forwarded to all 10 subscribers.

The 'subscribe' action registers a recipient, such as an email address, SMS phone number, Amazon SQS queue or AWS Lambda function. Then, when a message is sent to the topic, that recipient receives the message.

You say:

I will call the method publish to publish a message like "test message", which will output "Message test message sent to the topic testTopic" on successful publish. I expect the subscriber to then respond with "this is subscribe response: test message" , but it doesn't. It only fires when the subscriber is first initialized on server start with this message...

I'm not sure what you mean by "expect the subscriber to respond" -- the act of subscribing is done only once, to register the recipient. Then, each message will be delivered to the recipient (eg SMS phone message or HTTP endpoint). So, it depends on what you registered as the recipient as to whether you would expect a "response".

Unfortunately, the code in your Question does not show any Subscribe parameters, so it is difficult to say what the expected behaviour should be.

Thank you so much for this answer! Your explanation of subscribe clears it up for me. Now I see that the responsibility of subscribe function is just making sure that the SQS or whatever protocol is subscribed and connected to the topic and nothing more. If I want to interact with the queue itself I can't do it through SNS but have to interact directly. — Mike Yim, Nov 05 '22 at 01:42

SNS publishes to the SQS queue but the SNS subscription is not consuming the messages published to the SQS queue

1 Answers1