1

I am trying to retrieve all buckets s3 via boto3 python. I have tried;

s3_resource.buckets.all() and s3_client.list_buckets()

both of these gives at most 1000 buckets but there are over 1000 buckets that I've found. Is there a way to get all buckets?

I have also seen Java and C++ use an iterator to traverse through the list, is there something similar for python?

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_buckets

Using Paginator for list_buckets

paginator = self.client.get_paginator('list_buckets')
pages = paginator.paginate()

OperationNotPageableError: Operation cannot be paginated: list_buckets

About writing a custom paginator

list_buckets doesn't take in any request parameters. Even the API call docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html However in the response I see transfer-encoding : chunked

I wonder if I can use that somehow?

static const
  • 953
  • 4
  • 16
  • similar question : https://stackoverflow.com/questions/54314563/how-to-get-more-than-1000-objects-from-s3-by-using-list-objects-v2 – saranjeet singh Jul 28 '20 at 09:18
  • @saranjeetsingh Thanks for your response. however, in the question you posted they are trying to list the objects in a bucket. I am trying to list all the buckets that are available – static const Jul 28 '20 at 14:19
  • have you tried to create custom paginator? can you paste the exact error. – saranjeet singh Jul 28 '20 at 14:44
  • So I am not getting an error. It's just that I am limited to 1000 buckets (which makes me believe there is a limitation). I'll edit my question with the error in paginator – static const Jul 28 '20 at 14:59
  • no paginator is available for list buckets. have to write custom one. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#paginators – saranjeet singh Jul 28 '20 at 15:02
  • Aah I see what you mean. `list_buckets` doesn't take in any request parameters. Even the API call https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html However in the response I see `transfer-encoding : chunked`. Would you happen to know a good way to get the next set of results for this? – static const Jul 28 '20 at 15:10
  • 1
    `transfer-encoding : chunked` is not relevant here. If `s3_resource.buckets.all()` and `s3_client.list_buckets()` would return string token called next_token in response then we can write custom paginator. next_token just tells us that there are more results are available to fetch. – saranjeet singh Jul 28 '20 at 15:51
  • Gotcha, thank you so much for all the help! – static const Jul 28 '20 at 19:59

0 Answers0