1

In short, I am interested in building loosely coupled Microservices connected via SNS (for the most part) in order to process API requests in real-time.

Premise

  • Need all of this to occur within a single POST request response body
  • Cannot ask the client to pull for success upload and/or successful routing.

AWS API Gateway Endpoints

  • POST /api/documents/uploadAndRouteDownWorkflow, executes documents.upload and receives a combined response from documents.upload and workflows.routeDocument functions indicating full success (upload and route worked), partial success (upload but not route), or complete failure (upload failed)

Lambda Functions (executed in order):

- documents.upload

  • Invoked from API Gateway endpoint
  • Uploads documents to a DMS (Document Management System)
  • Creates a SNS message to a workflow microservice in order to route document

- workflows.routeDocument

  • Invoked from subscribed SNS topic
  • Routes document/documents in SNS message
  • Returns a success/failure to the original api request

Caveats why documents.upload does not invoke worksflows.routeDocument internally

  1. Microservices not loosely coupled anymore
  2. Double compute time for both lambda functions if forcing to be synchronous (is it possible )

Is this pattern possible?

Thanks!

David Garza
  • 161
  • 1
  • 9

1 Answers1

1

Here's where it breaks down:

-workflows.routeDocument

  • Returns a success/failure to the original api request

That isn't possible. By using SNS you have decoupled the services to the point that the Lambda function responsible for generating a response to the API Gateway request (documents.upload) has no idea what has happened in the other Lambda function. The workflows.routeDocument Lambda function has no access to the API Gateway event and context objects and thus is not able to update the API response.

The only way this would work is if the Lambda function invoked by API Gateway did some sort of polling to wait until the other function invocation was complete, and then somehow accessed the return state (stored in a database or something?) and returned that in the response. I think that's going to introduce a lot of latency in your request handling.

In this instance I think it makes more sense for documents.upload to invoke the workflows.routeDocument Lambda function directly.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • So if I understand correctly, I will need to maintain some type of synchronous flow in order to return a result indicating success failure to the original POST request? On this, if I want to keep these pieces loosely coupled, I would need to look to a different solution outside of AWS it seems. Reading up on RPC pattern within RabbitMQ via Direct Reply-to (https://www.rabbitmq.com/direct-reply-to.html) seems to be in line with what I am interested in doing). Is there another way in AWS to keep things loosely coupled while maintaining synchronous behavior? – David Garza Feb 13 '17 at 14:31
  • The RabbitMQ RPC pattern could be implemented with Amazon's SQS also. However you need to realize that RabbitMQ is still using queues and polling to communicate a response. In any fully decoupled solution you are going to have a long-running `documents.upload` function polling a database or queue or something waiting for a response that is generated by the `workflows.routeDocument` function. – Mark B Feb 13 '17 at 14:50
  • @DavidGarza You might be interested in today's AWS annoument: https://aws.amazon.com/about-aws/whats-new/2017/02/amazon-api-gateway-integration-with-aws-step-functions/ – Mark B Feb 15 '17 at 20:36
  • Great Tip! It certainly appears to be on par with what I'm looking to do. – David Garza Feb 16 '17 at 02:21