How to access dictionary's values in a dataframe containing dynamodb table with dictionary structure?

Question

I have a dataframe like this for my table in dynamodb:

col1             col2               col3               col4
{'S': 'miyuki'}  {'S': 'space'}    {'S': 'shape'}    {'S': 'blue'}
{'N': '22'}      {'S': 'dfgh'}     {'N': '123'}      {'S': 'red'}
{'S': 'normal'}  {'S': 'this'}     {'S': 'with'}     {'S': 'list'}

I just want values like this:

col1             col2               col3               col4
miyuki           space              shape              blue
22               dfgh               123                red ...

Also, if I use a resource instead of client the link here, it says that all these are not needed, but I could not apply the resource method to my own code. My code (source):

import boto3
import pandas as pd
from ast import literal_eval
client = boto3.client('dynamodb')

def dump_table(table_name):
    results = []
    last_evaluated_key = None
    while True:
        if last_evaluated_key:
            response = client.scan(
                TableName=table_name,
                ExclusiveStartKey=last_evaluated_key
            )
        else: 
            response = client.scan(TableName=table_name)
        last_evaluated_key = response.get('LastEvaluatedKey')
        
        results.extend(response['Items'])
        
        if not last_evaluated_key:
            break
    return results

# Usage
data = dump_table('my_table_name')
df=pd.DataFrame(data)

and i tried this:


data = dump_table('my_table_name')
df=pd.DataFrame(data)
s=pd.DataFrame()
a=df.columns.to_list()
for i in a:
    
    s[i] = pd.json_normalize(df[i])

but, I am getting this error:

ValueError: Cannot set a frame with no defined index and a value that cannot be converted to a Series

score 0 · Answer 1 · answered Oct 11 '21 at 12:34

By default, that's the way DynamoDB returns the data (it has its own version of JSON). To convert it to regular JSON, you can use a package named dynamodb_json:

from dynamodb_json import json_util
data = json_util.loads(dump_table('my_table_name'))

With that, data will be regular JSON, so you will be able to use it the way you wanted

How to access dictionary's values ​in a dataframe containing dynamodb table with dictionary structure?

1 Answers1

How to access dictionary's values in a dataframe containing dynamodb table with dictionary structure?