0

The challange

I would like to build a simple and fast (< 50ms) API with API gateway that works as an intelligent cache. The request to the API should batch fetch some items from DynamoDB based on their keys. However, if one or more items are not found in dynamoDB I would like to somehow trigger a lambda function, who knows how to populate those missing items for later requests) in dynamoDB.

My idea

The way I though of doing this is by creating any missing items in DynamoDB only with their key and then use DynamoDB steams to invoke a lambda function who reads the records and fill's their blank attributes with values from another external API.

Problem

My problem is however, that as far as I can figure out I can ONLY do ONE request to dynamoDB (either a BatchGetItem or a conditional PutItem) per request to API gateway. Is there some way of achieving what I want.

I would really like to use API gateway as this is needed to scale quite aggressively and I would rather not handle the scaling of servers and applications.

Alternatives

Also an alternative would be to have the client code make the decision of what items were missing from the response and then send a second request to "queue" those for population, however I would really like that logic to NOT go on the client side and and also to avoid two network roundtrips.

I also looked into if dynamoDB would be able to do this "find or create"-behaviour for me - but as it seems, no luck: What event can be triggered to fire a lambda function in DynamoDB?

Niels Kristian
  • 8,661
  • 11
  • 59
  • 117

1 Answers1

0

If your API Gateway endpoint calls a Lambda function (instead of doing a DynamoDB proxy which wouldn't be a clean API anyway) then you can run whatever code you want. You can make as many calls to DynamoDB and other services as you want whenever an API Gateway call triggers the Lambda function. The only restriction is the amount of time your Lambda function can run.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • But wouldn't that be very slow? Having to wait for Lambda to spin up and run code? I just added a requirement of < 50ms response time of the API – Niels Kristian Oct 25 '17 at 16:06
  • If you use Python or NodeJS then the Lambda function won't take very long at all to spin up. And if the function receives enough traffic to stay warmed, then you won't incur the startup time again. – Mark B Oct 25 '17 at 16:08
  • I have no experience with lambda. So what latency would I expect from lambda it self (not the service calls to DynamoDB)? – Niels Kristian Oct 25 '17 at 16:10
  • So after a cold start your function will be deployed and ready to accept new requests. At that point it will be like kicking off a script locally on your computer, it will run in milliseconds. – Mark B Oct 25 '17 at 16:15
  • What does "in milliseconds" mean? 2, 5, 10, 50, 100, 500? – Niels Kristian Oct 25 '17 at 16:16
  • 1
    Obviously that depends on the content of the script, what the script is doing. I'm not going to give you some number and tell you that's how long your script will take to run. It would take you less time to test it out than it would to continue the discussion. – Mark B Oct 25 '17 at 16:19
  • Sure, I was not talking about the execution time of the script it self, but only the lead time for lambda to get ready. It was my understanding, that lambda was not made for live requests but only background processing due to latency of the service, but maybe I'm wrong? – Niels Kristian Oct 25 '17 at 20:44
  • That's definitely not a correct understanding. People are building websites running on Lambda. You can use Lambda to intercept and modify HTTP responses in CloudFront. API Gateway + Lambda is the most common and popular way to use API Gateway to serve APIs. API Gateway would be an almost worthless service if Lambda were slow. – Mark B Oct 25 '17 at 21:18
  • Good news. Thanks for the response! – Niels Kristian Oct 26 '17 at 06:46