1

I'm looking for the best approach as to how I can go about doing validation of a message as its enqueued in async messaging based systems.

Scenario: Let's say we have a two services A and B where they need to interact with each other asynchronously. And we have a queue between them lets say SQS which will receive the message from A, which will be then polled by service B.

Ask: How can I validate the message like doing schema validation as its enqueued to SQS since currently SQS doesnt have any in-built schema validation functionality like we have for JMS

Couple of options I can think of:

  1. Have a validation layer maybe a small service sitting between A and SQS queue but not sure how feasible this will be
  2. Use some sort of MOM like AWS Eventbridge between A and SQS queue as it has functionalities to validate schemas as well as it could act as a central location to store all the schemas
  3. Have a rest endpoint in B that'll do the validation and have SQS sitting behind B but then this removes the async communication b/w A and B

Would appreciate any inputs on the above ask and how it could be resolved via best practices.

idanz
  • 821
  • 1
  • 6
  • 17
Vibhu Tiwari
  • 11
  • 2

2 Answers2

0

I'd recommend to read about the Mediator Topology of Event-Driven architecture style. From the details that you shared, it sounds to me that putting a "Mediator Service" called M for example, which will get messages from A, make the required validations, and then will send the message to SQS on its way to B - will achieve what you want.

idanz
  • 821
  • 1
  • 6
  • 17
  • Thanks I basically followed the rest pattern where first the message will be validated via rest endpoint and if its success then it'll be published to the queue. So queue is moved behind service B instead of having between A and B. And yes publisher and consumer will be running in parallel in service B – Vibhu Tiwari Jun 14 '22 at 05:56
0

Validation of the message payloads can occur on the "way in" or the "way out" depending on your use case and scaling needs. Most scenarios will aim to prevent invalid data getting too far downstream i.e. you will validate before putting data into SQS.

However, there are reasons you may choose to validate the message payload while reading from the queue. For example, you may have many services adding messages, those messages may have multiple "payload versions" over time, different teams could be building services (frontend and backend) etc. Don't assume everything and everyone is consistent.

Assuming that the payload data in SQS is validated and can be processed by a downstream consumer without checking could cause lots of problems and/or breaking scenarios. Always check your data in these scenarios. In my experience it's either the number one reason, or close to it, for why breaking changes occur.

Final point: with event-driven architectures the design decision points are not just about the processing/compute software services but also about the event data payloads themselves which also have to be designed properly.