0

I am listing all the available versions of a particular key on s3. Two issues:

  1. The first 1000 version ids that I obtain are not valid "NoSuchVersion". The remaining version ids work fine.
  2. The IsTruncated flag never goes back to false once I loop back.

Issue #2 isn't a big deal for me. But #1 is a show stopper.

Here is a code snippet:

Model::ListObjectVersionsRequest object_request;
object_request.WithBucket(this->bucket_name);
object_request.WithKeyMarker( ... + ".json");
do {
    auto list_versions_outcome = this->s3_client->ListObjectVersions(object_request);
    //...
    //prepare for next iteration
    object_request.SetVersionIdMarker(list_versions_outcome.GetResult().GetNextVersionIdMarker());
} while (keep_looking);

thoughts?

jarmod
  • 71,565
  • 16
  • 115
  • 122
user8993519
  • 1
  • 1
  • I don't understand `object_request.WithKeyMarker( ... + ".json");` Also, don't you need to set both the versionid and next key marker to iterate correctly? I think you are assuming ListObjectVersions always returns versions of the exact key you asked for and that may not be what's happening -- examine the key of each element, not just the versionid. Tell us what you find. – Michael - sqlbot Nov 23 '17 at 05:52
  • Hi Michael, thanks for the comment. the .WithKeyMarker(... + ".json") line is to specify the key that I wish to list all versions of. I'm trying to list all the versions of just one key in that bucket. Since that key is not changing, I don't need to re-set it every iteration. – user8993519 Nov 27 '17 at 18:14
  • You're making some assumptions, and not using this exactly as intended. You need to look at the key you're being given with each element of the response, because it may not be what you expect. If you want to list the versions of one specific key, set it as the prefix, not as the key marker for the first request. Then set the key and version markers on each subsequent request, even if it seems unnecessary. Markers are for continuation. They can be used in the first request, but the results may not line up with your expectations. You should also set the delimiter to `/` to avoid ambiguities. – Michael - sqlbot Nov 27 '17 at 18:28

1 Answers1

0

So I found a workaround for this issue. The idea is to limit the size of the first page of version id results to just one. After that is done, we can bring the size of the pages back to 1000. Even if the first result gets the wrong version id, this does not matter since retrieving the most recent version is the default behavior of s3.

The code becomes:

    Model::ListObjectVersionsRequest object_request;
object_request.WithBucket(this->bucket_name);
object_request.WithKeyMarker( ... + ".json");
object_request.WithMaxKeys(1);//HAX because of the first 1000 issue
do {
    auto list_versions_outcome = this->s3_client->ListObjectVersions(object_request);
    //...
    //prepare for next iteration
    object_request.SetVersionIdMarker(list_versions_outcome.GetResult().GetNextVersionIdMarker());
    object_request.SetMaxKeys(1000); // reset the page number to 1000
} while (keep_looking);
user8993519
  • 1
  • 1