Unable to Parse JSON file from S3 in AWS Quick Sight

Question

I developed a pipeline in AWS where I collect CPU Temperature from my notebook via Python 3, send to AWS IoT Core using security protocols, a rule sends data to Cloud Watch and DynamoDB saves it. A Data Pipeline was created to save this DynamoDB data to S3 and I want to generate plots of this data with Quick Sight.

However, I am not being able to make Quick Sight properly read the file. The S3 file looks like this:

{"timestamp":{"s":"1526819850637"},"payload":{"m":{"$Temperature":{"s":"42.000"}}}}
{"timestamp":{"s":"1526819976032"},"payload":{"m":{"$Temperature":{"s":"42.000"}}}}
{"timestamp":{"s":"1526819934216"},"payload":{"m":{"$Temperature":{"s":"42.000"}}}}
{"timestamp":{"s":"1526817845094"},"payload":{"m":{"$Temperature":{"s":"48.000"}}}}

When I use the manifest file below, Quick Sight successfully reads data but the 'parseJson' command in New Calculated field disappears, making impossible to read the JSON:

{
    "fileLocations": [
        {
            "URIs": [
                "https://s3.amazonaws.com/my-bucket2/2018-05-20-12-32-49/12345-a279-1234-2269-491212345"
            ]
        },
        {
            "URIPrefixes": [
                "https://s3.amazonaws.com/my-bucket2/2018-05-20-12-32-49/12345-a279-1234-2269-491212345"
            ]
        }
    ],"globalUploadSettings": {
        "format": "CSV","delimiter":"\n","textqualifier":"'"
    }

}

Quick Sight reads the JSON as:

{{"timestamp":{"s":"1526819850637"},"payload":{"m":{"$Temperature":{"s":"42.000"}}}}}

... with no 'parseJson' command.

Data in Quick Sight has no missing values and the AWS pipeline works perfectly. What can I do ?

Answer 1

I found the way to make it work. Simply:

{
    "fileLocations": [
        {
            "URIs": [
                "https://s3.amazonaws.com/your-bucket2/2018-05-20-12-32-49/123456789"
            ]
        },
        {
            "URIPrefixes": [
            "https://s3.amazonaws.com/your-bucket2/2018-05-20-12-32-49/123456789"
            ]
        }
    ],"globalUploadSettings": {"format":"JSON",
    "delimiter":"\n","textqualifier":"'"
    }
}