DynamoDB record not added through map loop

Question

I am fetching prices (usually thousands) from an API using Axios and then want to store it in DynamoDB. If I invoke the lambda function locally everything works as expected, but if I deploy the function and call it using AWS CLI it does not store any values in DynamoDB anymore. The data I receive in the request and also the response from the Axios call is the same.

I somehow think it is a scope issue of the async function calling DynamoDB, but I am not able to solve it. Looking forward to your suggestions. Let me know if you need more code.

updatePrice.js

import { updatePrice } from "./libs/pricing-lib";
import {
    success,
    failure
} from "./libs/response-lib";

export async function main(event, context) {
    try {
        let result = await updatePrice(event.pathParameters.id, event.pathParameters.date);
        return success(result);
    } catch(e) {
        return failure(e);
    }
}

dynamodb-lib-js

import AWS from "aws-sdk";
export function call(action, params) {
    const dynamoDb = new AWS.DynamoDB.DocumentClient();
    return dynamoDb[action](params).promise();
}

pricing-lib.js

export async function updatePrice(stockid, from) {
  try {
    let url = getUrl(stockid, from);
    const resultEodApi = (await axios.get(url)).data;

    resultEodApi.map((price) => {
      try {
        let priceParams = {
          TableName: process.env.pricesTableName,
          Item: {
            stockid: stockid,
            date: price.date,
            close: price.close
          }
        };
        dynamoDbLib.call("put", priceParams);
      } catch (e) {
        return e;
      }
    });

    return true;
  } catch (e) {
    return e;
  }
}

TableName: process.env.pricesTableName looks fishy. I would double check that reference in your YAML file and see what happens, unless that's what you've called your table. — lopezdp, Dec 20 '19 at 06:09
That’s just a weird way to name the table. I think if this would be wrong also the locally invoked function wouldn’t be working, right? — sebastiansieber, Dec 20 '19 at 06:48

score 5 · Accepted Answer · answered Dec 26 '19 at 09:19

Expanding a little on Ashish's answer.

The problem:

As Ashish said, the reason the put operation does not work in the Lambda deployment is that the call is performed asynchronously.

The call to dynamoDb[action](params).promise() starts the asynchronous put operation, and returns a Promise object. When the put operation returns, the promise will be resolved.

However, in your code you neither await for the promises to be resolved, nor do you return the promises as an output from the handler. The call to updatePrice terminates and returns undefined, at which point AWS Lambda suspends the execution of the function. As a result, the put calls never go through.

Why is there a difference between the local and remote executions?

The reason you see a difference between the remote and local executions is that a local node.js process and a Lambda function have different semantics.

When you run a node.js process locally, the node.js process only terminates once all promises have been resolved¹.

The Lambda execution behaves differently. Lambda does not wait for promises to be resolved before it terminates the execution². Instead, Lambda terminates the execution as soon as it resolves the promise that is returned by the handler. In your case, the handler function returns true³, so it is resolved immediately, and the execution terminates. At that point in time, your put calls haven't resolved yet.

To illustrate the difference, consider the following code:

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}       

module.exports.hello = async (event) => {
  sleep(300).then(() => console.log("Finished doing my asynchronous thing, can exit now!"))    
  console.log("Function execution done!")
  return true;
};

// call the handler when running locally:
if (require.main === module) { module.exports.hello(); }

When you're running this locally, the function execution finishes, but the node process is still running until the promise is resolved. Running locally, you will get the following output (notice the order):

> Function execution done!
> Finished doing my asynchronous thing, can exit now!

Running on lambda, when the function execution ends, so does the rest of the run. The promise is never resolved. We will get the following output:

START RequestId: <id> Version: $LATEST
2019-12-26T09:04:28.843Z    <id>    INFO    Function execution done!
END RequestId: <id>
REPORT RequestId: <id>  Duration: 3.37 ms   Billed Duration: 100 ms Memory Size: 1024 MB    Max Memory Used: 71 MB  Init Duration: 114.44 ms

We only get the Function execution done! print. Notice also that the execution Duration is only 3.37ms. The 300ms sleep we ran asynchronously did not have time to resolve before AWS stopped the process.

Fixing the problem

From the AWS Lambda developer guide:

If your code performs an asynchronous task, return a promise to make sure that it finishes running. When you resolve or reject the promise, Lambda sends the response or error to the invoker.

You can use Ashish's solution -- returning the promises you created. Alternatively, you can explicitly await the Promise.all. In either case it is important to make sure that you do not lose any of the promises you create before returning from function.

¹ More specifically, it waits for the event loop to be empty.

² It actually suspends the execution. The execution will continue from the same place the next time you invoke the handler. If you made multiple consecutive calls to the handler, some of your puts would probably get through, depending on timing, and on how well the aws-sdk library handles these kinds of interruptions in the execution flow, but it is very hard to predict.

³ Actually, the return value of an async function is always a promise that wraps the return value. So in your case what you have is a return true; -- the true is wrapped in a Promise object, which is resolved immediately and the execution terminates.

score 0 · Answer 2 · answered Dec 20 '19 at 09:02

The problem lies in pricing-lib.js as you are doing async operation inside map. You will need to return promise from inside the map and wrap map in promise all. see here

export async function updatePrice(stockid, from) {
  try {
    let url = getUrl(stockid, from);
    const resultEodApi = (await axios.get(url)).data;

    return Promise.all(resultEodApi.map((price) => {
      try {
        let priceParams = {
          TableName: process.env.pricesTableName,
          Item: {
            stockid: stockid,
            date: price.date,
            close: price.close
          }
        };
        return dynamoDbLib.call("put", priceParams);
      } catch (e) {
        return e;
      }
    }));

    return true;
  } catch (e) {
    return e;
  }
}

Thanks a lot for looking into my code. Now I get a 502 Bad Gateway error... any pointers what to look for? — sebastiansieber, Dec 20 '19 at 09:28
And by the way, again, if I locally invoke the new lambda function (with Promise.all etc) it WORKS, but not if I use AWS CLI. — sebastiansieber, Dec 20 '19 at 09:42

score 0 · Answer 3 · answered Nov 30 '21 at 11:49

For this problem, I think it is better to use batch requests.

https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/dynamodb-example-table-read-write-batch.html

I also have the same problem and I rewrote my util function to use batch requests, much easier to understand now.

DynamoDB record not added through map loop

3 Answers3

The problem:

Why is there a difference between the local and remote executions?

Fixing the problem