boto3 searialisation outputs strange format

Question

I am writing a Lambda Python function on AWS. It retrieves a DynamoDB item and I want to return that back to the caller in a JSON format. If I don't serialise the item, there are errors with Python's json.dump function.

from boto3.dynamodb.types import TypeDeserializer, TypeSerializer
def serialize(dynamo_obj: dict) -> dict:
    serializer = TypeSerializer()
    return {
        k: serializer.serialize(v)
        for k, v in dynamo_obj.items()
    }

table = dynamodb.Table(os.getenv('STORAGE_NAME'))

response = table.get_item(Key={'id': some_id})
item = response.get('Item', None)

if item:
    return {
        'statusCode': 200,
        'headers': { 'Content-Type': 'application/json' },
        'body': json.dumps(serialize(item))
    }

However, when I serialise this way, the returned item contains some strange extra keys (some 'N' and 'S' keys that they don't appear in the dynamodb table).

What's the right way to make the dynamodb item compatible with JSON format so it can be returned back to the caller?

Related question [here](https://stackoverflow.com/questions/36558646/how-to-convert-from-dynamodb-wire-protocol-to-native-python-object-manually-with). Note: `N` and `S` relate to the attribute type (number and string). — jarmod, May 26 '22 at 20:38
I see, but isn't there a way to make that item Python compatible and keep it in it's right format? I just really want to get item from dynamodb, and return it back to the REST API caller. — Phrixus, May 26 '22 at 20:40
Allegedly you can use the TypeDeserializer (see [here](https://stackoverflow.com/questions/48866278/wrap-unwrap-dynamo-db-streams)). — jarmod, May 26 '22 at 20:42
I have tried that but it gives a different error: "AttributeError: 'str' object has no attribute 'keys'" — Phrixus, May 26 '22 at 20:45
Not sure what your code looks like that's triggering that error. — jarmod, May 26 '22 at 21:03

score 0 · Answer 1 · answered Jun 16 '22 at 10:10

This is a common enough pattern that I wrote a library to perform the conversion. You can use cerealbox as follows:

from cerealbox.dynamo import from_dynamodb_json

# converts from dynamodb json to regular python dictionary
payload = from_dynamodb_json(item)

DynamoDB saves numeric values as an instance of decimal.Decimal which is not json serializable. This will result in json.dumps throwing an exception. There is another utility that can be used to make sure that all datatypes are converted to a json serializable type. In this case it will convert Decimal to string so that precision is preserved. This can be hooked into as follows:

from cerealbox.jsonable import as_jsonable

# option 1 - create a dict thats json serializable
json_serializable_payload = as_jsonable(payload)


# option 2 - use directly with json.dumps
body = json.dumps(payload, default=as_jsonable)

cerealbox can be extended to customize the behaviour should you require Decimal's to be serialized as floats or integers.

boto3 searialisation outputs strange format

1 Answers1