1

I have a system where I publish updates to a shared topic meant for specific consumers. I noticed messages getting stuck in the queue due to a lack of selective listening in SQS consumers, so messages are being hijacked.

Example:

Given: Message{destination: A, payload: 1234}

Given: ConsumerA, & ConsumerB

I expect Message to be processed by ConsumerA. However, it gets hijacked by Consumer B continuously. It receives the message, then refuses to process it since the destination field doesn't match, leading to the visibility timeout to expire, and the message put back on the queue.. but due to the nature of SQS, ConsumerB has an equal chance of picking the message again.

My question is, what patterns are used to solve this type of issue? I'm considering creating a queue per consumer but it has drawbacks specific to the system im working on.

If I could only listen for messages with matching attributes, problem solved, but that's seemingly not the case.

Is there any other way?

Loc12342
  • 23
  • 1
  • 4
  • Look at EventBridge that lets you receive, filter, transform, route, and deliver events. Or use an SNS topic to which all consumers subscribe and each consumer ignores messages not intended for it. – jarmod Oct 19 '22 at 23:58
  • @jarmod the SNS approach seems attractive. The messages im sending are FIFO, and it seems it can handle it now. Thanks for this, dunno how i overlooked it. – Loc12342 Oct 20 '22 at 04:28

1 Answers1

1

Sharing a single Amazon SQS queue is not an appropriate architecture for your use-case.

If you want your consumers to be able to 'request' a message from a particular subset, you should either use separate SQS queues or use a database. You could even store objects in Amazon S3 as a form of noSQL database.

Having consumers grab messages and then 'send them back' to the queue is not compatible with the design of the Amazon SQS service.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470