3

I am planning to develop an Event-Driven Microservices.

  1. I create a protobuf project, which defines several types of messages.

    • EmployeeMessage
    • UserMessage
    • ProcessMessage
    • ApplicantMessage

  2. Then, I compile the protobuf project to different languages, e.g. Ruby, Golang.

  3. Then, the upstream application will push the following type of events to the SNS, the SNS fanout message to multiple SQS, which owned by the different downstream consumers.

  4. Then, the downstream application consumes messages from SQS.

Here is a diagram to show the whole architecture.

enter image description here

When implementing it, I realize there is no way. Protobuf messages of different types are posted to SNS, the consumer doesn't know the type of each message and is not able to decode them.

Questions

  1. How do you implement your Event-Driven microservices? Does each type of message have its own SNS (stream)?

  2. Is there a way to allow me to push different message types to the same SNS (stream)? Do I need to append the message type in front of the payload?

Ryan Lyu
  • 4,180
  • 5
  • 35
  • 51

2 Answers2

1

In your protobuf schema, design the message to be sent to SNS like so:

message Event {
  oneof request {
    EmployeeMessage employee_message = 1;
    UserMessage user_message = 2;
    // etc
  }
}

Then have your receivers decode Event and check for the correct message type.

I think this answers both of your questions.

CJ Zeiger
  • 43
  • 3
0

A couple of options:

  1. Use a composition of SNS/SQS Topic for each targeted language. In your application building the protobufs, manage a mapping in SSM/AppConfig. Pros: Reduce Costs, SNS/SQS both charge for messages handled. Cons: Increase infrastructure wrangling. The map generation is easy to do in CFN automatically. Example: RubySNS/RubySQS --> RubyApp

  2. Store an attribute in the protobuf or use SNS Message Attributes to signal language destination. Message attributes may be cleaner for all SQS workers. Pro: Aligns with your current design if there are constraints forcing this. Cons: Requires the downstream workers to have some knowledge of the message.

Marcus DeAnda
  • 1