Exporting data from dynamo db to a csv file

Question

We want to export data from dynamo db to a file. We have around 150,000 records each record is of 430 bytes. It would be a periodic activity once a week. Can we do that with lambda? Is it possible as lambda has a maximum execution time of 15 minutes?

If there is a better option using python or via UI as I'm unable to export more than 100 records from UI?

Looks like you want to use Data Pipeline https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-template-exportddbtos3.html https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-importexport-ddb-part2.html — DanielC, Jan 30 '20 at 15:42

score 3 · Answer 1 · answered Jan 30 '20 at 18:32

You can export your data from dynamodb in a number of ways.

The simplest way would be a full table scan:

dynamodb = boto3.client('dynamodb')

response = dynamodb.scan(
    TableName=your_table,
    Select='ALL_ATTRIBUTES')

data = response['Items']

while 'LastEvaluatedKey' in response:
    response = dynamodb.scan(
        TableName=your_table,
        Select='ALL_ATTRIBUTES',
        ExclusiveStartKey=response['LastEvaluatedKey'])

    data.extend(response['Items'])

# save your data as csv here

But if you want to do it every x days, what I would recomend to you is:

Create your first dump from your table with the code above.

Then, you can create a dynamodb trigger to a lambda function that will receive all your table changes (insert, update, delete), and then you can append the data in your csv file. The code would be something like:

def lambda_handler(event, context):
    for record in event['Records']:
        # get the changes here and save it

Since you will receive only your table updates, you don't need to worry about the 15 minutes execution from lambda.

You can read more about dynamodb streams and lambda here: DynamoDB Streams and AWS Lambda Triggers

And if you want to work on your data, you can always create a aws glue or a EMR cluster.

score 3 · Answer 2 · answered Feb 02 '20 at 11:40

One really simple option is to use the Command Line Interface tools

aws dynamodb scan --table-name YOURTABLE --output text > outputfile.txt

This would give you a tab delimited output. You can run it as a cronjob for regular output.

The scan wouldn't take anything like 15 minutes (probably just a few seconds). So you wouldn't need to worry about your Lambda timing out if you did it that way.

score 1 · Accepted Answer · answered Feb 04 '20 at 09:02

Guys we resolved it using AWS lambda, 150,000 records (each record is of 430 bytes) are processed to csv file in 2.2 minutes using maximum available memory (3008 mb). Created an event rule for that to run on periodic basis. Time and size is written so that anyone can calculate how much they can do with lambda

score 0 · Answer 4 · answered Feb 02 '20 at 12:02

0

You can refer to an existing question on stackoverflow. This question is about exporting dynamo db table as a csv.

answered Feb 02 '20 at 12:02

Rahul Goel

842
1
8
14

Exporting data from dynamo db to a csv file

4 Answers4