AWS S3 C++ : List all the object when result is truncated

Question

I use following code to list all the keys in my S3 bucket.

I have around 15,000 objects in the bucket. However this code just loops through the first 1000 objects infinitly. Seems it doesnt respect the SetMarker() method.

Any tips on how to fix this ?

#include <aws/s3/S3Client.h>
#include <aws/s3/model/ListObjectsRequest.h>
#include <aws/s3/model/Object.h>

int main(int argc, const char* argv[])
{
    Aws::SDKOptions options;
    Aws::InitAPI(options);
    {
        Aws::Client::ClientConfiguration config;
        config.region="ap-northeast-1";

        Aws::S3::S3Client s3_client(config);
        Aws::S3::Model::ListObjectsRequest objects_request;

        objects_request.WithBucket("MYBUCKETNAME").WithPrefix("some-prefox");

        bool isDone = false;
        bool isFailed= false;
        Aws::S3::Model::ListObjectsOutcome outcome;
        int c=0;

        while(!isDone) {
                 outcome=s3_client.ListObjects(objects_request);
                 if(!outcome.IsSuccess()) break;
                 //process
                 Aws::Vector<Aws::S3::Model::Object> object_list = outcome.GetResult().GetContents();

                 for (auto const &s3_object : object_list)
                {
                    std::cout << "* " << s3_object.GetKey() << std::endl;
                    c++;

                }
                std::cout<<"--------- Break"<<c<<"\n";
                 isDone=!outcome.GetResult().GetIsTruncated();
                 if(!isDone) {
                    objects_request.SetMarker(outcome.GetResult().GetNextMarker());

                }
            }
            std::cout << "Count "<<c<<"\n";
            //check isFailed
            if(!outcome.IsSuccess()) {
                std::cout << "ListObjects error: " <<
                outcome.GetError().GetExceptionName() << " " <<
                outcome.GetError().GetMessage() << std::endl;
            }

    }

    Aws::ShutdownAPI(options);
}

score 3 · Accepted Answer · answered Oct 03 '18 at 15:51

3

According to the documentation for NextMarker:

Note

This element is returned only if you specify a delimiter request parameter. If the response does not include the NextMarker and it is truncated, you can use the value of the last Key in the response as the marker in the subsequent request to get the next set of object keys.

Your code should therefore be:

if(!isDone) {
  objects_request.SetMarker(outcome.GetResult().GetContents().back().GetKey());
}

Also note that the V1 ListObjects method is deprecated, you should be using ListObjectsV2 which uses explicit continuation tokens which are a bit easier to use.

answered Oct 03 '18 at 15:51

Alan Birtles

32,622
4
31
60

Birties : thanks for the tip.. it worked.. however ListObjectsV2 f ailed giving error "ListObjects error: InvalidArgument Unable to parse ExceptionName: InvalidArgument Message: The continuation token provided is incorrect" – Ashika Umanga Umagiliya Oct 04 '18 at 12:11
1

@AshikaUmangaUmagiliya I guess you aren't using it correctly, post a new question with your full code and I'm sure someone will tell you whats wrong. – Alan Birtles Oct 04 '18 at 12:12

AWS S3 C++ : List all the object when result is truncated

1 Answers1