1

We are building an application with a microservice architecture.

The microservice architecture will follow a message-oriented pattern, with AWS SQS.

We would like to return completion results from the consumer service back to the producer service.

This is the algorithm we are considering:

  1. Producer creates a message with a unique id
  2. Producer subscribes to a Redis channel that is named with the message id
  3. Producer places the message onto the SQS queue
  4. Consumer removes the message from the SQS queue and performs an operation
  5. Consumer publishes the results of the operation to the Redis channel that is named with the message id
  6. Producer recieves the completion results and resumes execution

Is this a reasonable way to pass message-completion results from a consumer back to a producer?

Joel Stevick
  • 1,638
  • 2
  • 16
  • 22
  • 1
    How about creating a response queue and consumer places all the responses on this queue? Producer and a listener threads running which will start further processing after getting response messages. This way you will avoid a queue for every message. – S.K. Aug 23 '18 at 11:34
  • I like that idea. Our application will have many producers at-scale, do you think that would be an issue for SQS, servicing (potentially) 100s of thousands of simultaneous listeners on a single (response) queue? Also, my impression was that Redis would be able to handle use of a channel for each message, do you think otherwise? – Joel Stevick Aug 23 '18 at 12:29
  • 1
    I think it wouldn't matter if you have thousands of channels or 1 channel with thousands of listeners. Both the scenarios will strain your broker the same way which can then be handled by clustering and scaling out. You can choose an approach by comparing ease of implementation in this case. – S.K. Aug 23 '18 at 13:49
  • My understanding is that you [cannot filter messages based on a message attribute in SQS](https://stackoverflow.com/questions/12630756/finding-certain-messages-in-sqs/12742463), this would seem to preclude the use of thousands of listeners on a single SQS queue, at-scale. Am I wrong about this? Or am I misunderstanding your suggestion? – Joel Stevick Aug 23 '18 at 14:04

1 Answers1

0

After continued research, it became apparent that message queues are not part of the solution. Point #5 in this article, "...or doesn’t even care about the result..." suggests (by implication) that we are simply using the wrong approach.

We changed our design so that request ordering is not important, and will make direct calls to AWS Lambda functions using the invoke api.

Joel Stevick
  • 1,638
  • 2
  • 16
  • 22