Manifest for multiple JSON files in S3

Question

I am trying to create a quicksight data source using a manifest for S3. The data at S3 is multiple JSON documents. Millions of them. I can create a dataset using a single document like this:

{
    "fileLocations": [
        {
            "URIs": [
                "https://s3-us-west-2.amazonaws.com/my-bucket/doc1.json"
            ]
        }
    ],
    "globalUploadSettings": {
        "format": "JSON"
    }
}

However, I would like ALL of the JSON documents. I am trying to do this without success:

{
    "fileLocations": [
        {
            "URIs": [
                "https://s3-us-west-2.amazonaws.com/my-bucket/*.json"
            ]
        }
    ],
    "globalUploadSettings": {
        "format": "JSON"
    }
}

How can you create a wildcard manifest for JSON documents on S3?

Hi Kevin, did you find any solution that really works? You talk about **millions** of jsons, but I only have **3000** jsons and all I get is an error importing the data set. It worked when I tested with less then **1000** jsons. The proposed solution is ok, but I want to know if it works with MILLIONS jsons. Thanks. — Costin, May 20 '19 at 08:22

score 3 · Answer 1 · answered Mar 21 '19 at 12:20

The following should get all files in a bucket

{
    "fileLocations": [
        {
            "URIPrefixes": [ 
                "s3://my-bucket/"          
            ]
        }
    ],
    "globalUploadSettings": {
        "format": "JSON"
    }
}

Manifest for multiple JSON files in S3

1 Answers1