1

We are building an application using microservices and AWS SQS for messaging.

We have a requirement for a single message to be handled by multiple microservices before being processed at a "final destination" microservice.

For example:

  • microservice A generates a message
  • the message must be validated by microservice B
  • the validated message must be transformed using microservice C
  • the transformed message must be applied to a data store using microservice D

Is there a pattern for dynamically (at message-creation time) prescribing the flow of a message between listeners on a message queue?

Joel Stevick
  • 1,638
  • 2
  • 16
  • 22
  • 1
    Perhaps this is an article which you find interesting: http://blog.arungupta.me/microservice-design-patterns/ – Stefan Aug 22 '18 at 11:43

1 Answers1

2

It seems selective routing of messages is not achievable with SQS as stated in the link below

Finding certain messages in SQS

As you can see from SQS docs https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/java/example_code/sqs/src/main/java/aws/example/sqs/SendReceiveMessages.java, there is no way you can add attributes to categorise messages i.e SQS is not architected for the use-case

The only possible solution is using three different queues for different types of messages and have each of the microservices listen to specific type of messages which is a typical use case of Pipe and filter design pattern

On the otherhand, if you consider building your own dynamic-routing queue, please refer this documentation to implement the content-based router as discussed in this enterprise design pattern document https://www.enterpriseintegrationpatterns.com/patterns/messaging/ContentBasedRouter.html

Hope it helps!

Aarish Ramesh
  • 6,745
  • 15
  • 60
  • 105
  • Your answer gets right to my question, thanks. What would you think of using a "dispatcher" microservice that listens for messages on a generic queue, inspects the message content and then relays the messages to the queues that are associated with the respective handler-microservices. And then have a separate "response" queue to which originators would listen for response messages? – Joel Stevick Aug 22 '18 at 12:30
  • 1
    @JoelStevick : The design you are suggesting is close what I was suggesting, the pipe and filter architecture pattern like below which uses separate SQS queues - service1 acts on the msg -> queues the msg to SQS Q1 , service2 listens for SQS Q1 and acts on it then pushes it to SQS Q2 , service3 listens for SQS Q3 and acts on it . Hope it's clear – Aarish Ramesh Aug 22 '18 at 13:35
  • 1
    So, if service3 wants to send a response message back to service1, how would that be implemented? I was thinking that a reasonable way would be to have a "response" queue to which service1 would listen to for completion results from any other service; would that be an anti-pattern? – Joel Stevick Aug 22 '18 at 14:24
  • From the answer I was able to identify the exact pattern that I was looking for, it is called "[routing slip](https://www.enterpriseintegrationpatterns.com/patterns/messaging/RoutingTable.html)" – Joel Stevick Aug 22 '18 at 20:03