3

I have a problem with retrieving the data from S3 using Amazon SDK. The problem is that it retrieves only 1000 elements, while indeed I have 10,000 elements in the aws_bucket_data -> currentDataDirectory. I do not use setMaxKeys(...), so the result seems to be weird.

BasicAWSCredentials credentials = new BasicAWSCredentials("...", "...");
        client = new AmazonS3Client(credentials);

ListObjectsRequest listObjectsRequest = new ListObjectsRequest()
                    .withBucketName(aws_bucket_data)
                    .withPrefix(currentDataDirectory);

ObjectListing objectListing = client.listObjects(listObjectsRequest);

System.out.println(objectListing.getObjectSummaries().size());

How can I solve this problem?

Dinosaurius
  • 8,306
  • 19
  • 64
  • 113

1 Answers1

6

AWS S3 API has a limit of maximum 1000 keys per response.

You will have to do multiple requests to retrieve all of your objects.

You can take a look at the API here:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html

I have found a example to retrieve all your objects:

How to list all AWS S3 objects in a bucket using Java

Community
  • 1
  • 1
kkflf
  • 2,435
  • 3
  • 26
  • 42
  • I would love to, but I am currently not at home, so I cannot provide you with more detailed code snippet. There are however a lot of topics about this subject, if you just google a bit. I would love to give you more details, but I cannot do that atm. – kkflf May 19 '17 at 12:55
  • I have added a stackoverflow link with an example – kkflf May 19 '17 at 13:02
  • 1
    I finally used this: `do { ... if (response.isTruncated()) { request.setMarker(response.getNextMarker()); } else { request = null; } } while (request != null);` – Dinosaurius May 19 '17 at 13:14