Lambda only inserts 4 records but iterates 248 times

Question

I have an AWS lambda (to be fired by a cloudwatch event) which should be pulling 248 records from one API and inserting 248 records into Dynamo DB. It successfully pulls 248 records, but the Dynamo table only contains 4 records once the execution finishes, with no errors.

This is the work horse code from the lambda:

public async Task<APIGatewayProxyResponse> Get(APIGatewayProxyRequest request, ILambdaContext context)
{
    try
    {
        var count = await LoadAutocomplete(Data);

        context.Logger.LogLine($"{count} items processed into table '{EnvironmentHelper.DynamoTableName}'");

        ...

        return response;
    }
    catch (Exception ex)
    {
        ...
    }
}

private async Task<int> LoadAutocomplete(IList<MatItem> data)
{
    var count = 0;

    var client = new AmazonDynamoDBClient();
    var table = Amazon.DynamoDBv2.DocumentModel.Table.LoadTable(client, EnvironmentHelper.DynamoTableName);

    foreach (var item in data)
    {
        var doc = new Amazon.DynamoDBv2.DocumentModel.Document();
        doc.Add("LANGUAGE", item.LANGUAGE);
        doc.Add("MAT_DESC", item.MAT_DESC);
        doc.Add("SALES_ORG", item.SALES_ORG);
        doc.Add("COUNTRY", item.COUNTRY);
        doc.Add("SPECIFICATION", item.SPECIFICATION);
        doc.Add("MATERIAL", item.MATERIAL);
        doc.Add("TEMPLATE_ID", item.TEMPLATE_ID);

        await table.PutItemAsync(doc);

        count++;
        System.Threading.Thread.Sleep(100);
    }

    return count;
}

And this is the output of the LogLine statement:

248 items processed into table 'my-data'

Here is the tf used to create the table:

# create the table
resource "aws_dynamodb_table" "prospectmaterials_table" {
  name              = "my-table"
  hash_key          = "MATERIAL"
  billing_mode      = "PROVISIONED"
  read_capacity     = 5
  write_capacity    = 5
  
  attribute {
    name = "MATERIAL"
    type = "S"
  }
}

What does the key schema of the table look like and how does the data look like you're trying to write to it? Maybe it's just writing to the same keys 248 times. — Maurice, Feb 26 '21 at 08:19
Crikey @Maurice you're right! The `hash_key` is a field `MATERIAL` and there are only 4 variations on that value. I have updated the code and included the tf. The problem I now face is that each record is unique, but only if all 7 fields are accounted for. How do I make DynamoDB take all the fields as (composite) key fields? — Matt W, Feb 26 '21 at 08:30
Short answer: You add an attribute called PK to which you add all the values that uniquely identify the item and use that as the partition/hash key. Longer answer: What you should do depends on what you want to do with the data. In DynamoDB you start with the access patterns and design your data model to fit these access patterns. That seems a little out of scope for this question though. Keep in mind that you want to design your keys in such a way that you can rely on `GetItem` or `Query` operations to fetch data and not `Scan`. — Maurice, Feb 26 '21 at 08:34

score 1 · Accepted Answer · answered Feb 26 '21 at 08:38

Your code looks right, so it's most likely the data and table design that are going to bite you.

In DynamoDB the primary key is what uniquely identifies an item and you need the primary key to write to/update an item. You're partition key (which is in this case also the primary key) is set to the MATERIAL attribute.

My guess is, that there aren't many distinct values for MATERIAL in the data and as a result of that your 248 writes result in four items, because they probably overwrite the existing items.

Thank you! I now have another error because I've re-assigned the hash_key of the table, but that's for another story... — Matt W, Feb 26 '21 at 08:44

Lambda only inserts 4 records but iterates 248 times

1 Answers1

Linked