S3/Cloudfront: how to list a folder's contents in a bucket

Question

I've written a method that signs URLs to read an S3 bucket that is accessed through Cloudfront:

private HTTPResponse signURL(String endpoint) {
    String keyPairId = 'keyPairId';
    String secret    = 'SOME_SECRET';
    String method    = 'GET';
    Datetime dt      = Datetime.now();
    Long l           = dt.getTime();
    Long expiryLong  = (l / 1000) + 3600;
    String expiry    = String.valueOf(expiryLong);
    String policy    = policy('http://' + cfhost + '/' + endpoint, expiry);
    String policyEnc = EncodingUtil.base64Encode(Blob.valueOf(policy));
    String rPolicy   = cfReplace(policyEnc);
    Blob mac         = Crypto.generateMac('HMACSHA1',
                                          Blob.valueOf(rPolicy),
                                          Blob.valueOf(secret));
    String signature = cfReplace(EncodingUtil.base64Encode(mac));

    HttpRequest req = new HttpRequest();
    req.setMethod(method);
    req.setEndpoint('http://' + cfhost + '/' + endpoint + '?Policy=' +
        policyEnc + '&Signature=' + signature + '&Key-Pair-Id=' +
        keyPairId);

    Http http = new Http();
    HTTPResponse res = http.send(req);
    return res;
}

Now, my method works fine if the endpoint is set to ''. If I put the folder name in the endpoint (like 00Q17000008LRcAEAW/*) I get a bunch of errors in the XML response.

XMLNode[ELEMENT,Error,null,null,null,[XMLNode[ELEMENT,Code,null,null,null,[XMLNode[TEXT,null,null,null,null,null,NoSuchKey,]],null,], XMLNode[ELEMENT,Message,null,null,null,[XMLNode[TEXT,null,null,null,null,null,The specified key does not exist.,]],null,], XMLNode[ELEMENT,Key,null,null,null,[XMLNode[TEXT,null,null,null,null,null,00Q17000008LRcAEAW/*,]],null,], XMLNode[ELEMENT,RequestId,null,null,null,[XMLNode[TEXT,null,null,null,null,null,some_req_id,]],null,], XMLNode[ELEMENT,HostId,null,null,null,[XMLNode[TEXT,null,null,null,null,null,some_host_id,]],null,]],null,]

I can't list the whole bucket because it's too big for the 1000-key limit. How can I list the folder?

I would strongly recommend using jets3t instead of hand-writing all this. — chrylis -cautiouslyoptimistic-, Jul 18 '16 at 01:51

score 3 · Accepted Answer · answered Jul 18 '16 at 11:17

See the S3 docs for List Objects, V1 and V2 for what S3 expects.

In short, you have to supply the prefix and delimiter (almost always /) in the query string, not the path. The path is always the root of the bucket, for object listings, e.g.:

https://dxxxexample.cloudfront.net/?delimiter=/&prefix=00Q17000008LRcAEAW/&...

This also means that if you have a default root object defined for the distribution, you can't also pull directory listings through the distribution. You'd need a second distribution or you'd need to fetch object listings directly from S3 in that case.

In any event, bear in mind that object listings are relatively expensive, over 10× the price of a GET request, so using CloudFront to fetch them might make sense if caching the responses makes sense, or you might want to do something entirely different, for object listings. I've been rolling out a setup using S3 event notifications, Lambda, and RDS, to store all the objects and their metadata in a separate near-real-time database, where I can get faster and more complete listings without the cost (and I can search objects based on their metadata).

Thanks for the in-depth information! That helps a lot. – t56k Jul 19 '16 at 01:42 — t56k, Jul 19 '16 at 01:42

S3/Cloudfront: how to list a folder's contents in a bucket

1 Answers1