0

Is there a way to implement producer/consumer messaging model in AWS where consumers are selective?

Consider a baggage carousel. Producer puts bags on a carousel and consumers pick up bags when they believe that they are theirs. Producer and the carousel itself don't know which bag belongs to whom. It's a consumer responsibility to pull the correct bag. Carousel personnel responsible only for removing unclaimed bags from the carousel and stowing them for further processing.

Can we have a message system in which multiple consumers are pulling the message in a self-service manner?

  • SQS does not allow consumers to be selective (having several picky consumers would all but guarantee that the message would go DLQ before reaching the intended one).
  • SNS and EventBridge require message filtering logic to be implemented/reproduced on server side, which means no "self service" for the consumers.

This messaging system, as a minimum, should allow sufficient time for all consumers to try to pull messages, and ideally, keep track if all consumers had actually checked the message before putting it to DLQ.

This messaging system should never let the message be consumed by more than one consumer.

AWS SQS Selective Polling Pattern

Can we filter messages from Amazon SQS queue by message attributes?

Alexander
  • 753
  • 2
  • 10
  • 25
  • There seems to be a conflict between your desire to "remove a message" (which implies a Queue) and the ability for consumers to "self service" by having selection logic in the consumers. As you state, SQS requires all consumers to be equal, while SNS effectively sends the message to all consumers (where they can "self service") but does not have the concept of a message being "taken off the carousel". You should go with the SNS/EventBridge method of sending messages to (potentially) all consumers and trusting that they will process the message as necessary. There should be no need for a DLQ. – John Rotenstein Aug 09 '23 at 23:08
  • @John Rotenstein With SNS/EventBridge, how do I avoid double processing, particularly if there are multiple instances of certain consumers? In "baggage carousel" model, consumers return bags to carousel themselves if, upon examination, they determine that the bag is not theirs. – Alexander Aug 09 '23 at 23:15
  • A Pub/Sub model should _encourage_ double-processing -- basically, any consumer that wants to process a particular type of message is welcome to do so. Your requirement that "Producer and the carousel itself don't know which bag belongs to whom" is contrary to Queue and Pub/Sub models. You'll need to develop your own custom solution for handling this, such as putting the messages into a database and letting them 'claim' messages. – John Rotenstein Aug 10 '23 at 00:36
  • I am not saying that I'm limited to Pub/Sub model. I do have an "exactly once" requirement though (I'm afraid I was not very clear about it). A message should be delivered exactly once to _any_ of the consumers (or to DLQ). – Alexander Aug 10 '23 at 05:17
  • Unfortunately, Amazon SQS does not support the requirement of "consumer responsibility to pull the correct bag". Rather, a consumer would need to pull a message and would be responsible for processing it rather than saying "it's not my bag". – John Rotenstein Aug 10 '23 at 08:29

0 Answers0