0

Is there any AWS sdk API that would give the total count of objects/keys available in a particular S3 bucket? with out using isTruncated() , listObjects() total iteration.

user12
  • 239
  • 2
  • 6
  • 18

2 Answers2

2

Not really. I think you have three options, depending upon your requirements right now:

  1. Iterating each object. The obvious answer, but perhaps the least favoured.
  2. Maintaining a running state of your own in something like DynamoDB. You could use a trigger on an S3 event to update your own values or use CloudWatch metrics.
  3. Use the S3 Inventory to get a list on a daily basis.

So, depending upon urgency, consistency requirements et al, these provide some options at least.

smcstewart
  • 2,016
  • 14
  • 16
-1

I know you have said without using listObjects() but may we know why you don't want to use it.

Is it because of the time taken to get all the objects data or is it because the method is asynchronous?

Anyway Amazon now recommends using the new listObjectsV2(params, callback) method. Maybe you can use that to get it.

Something like this

s3.listObjectsV2(params, function(err, data) {
    if (err) console.log(err, err.stack); // an error occurred
    else console.log(data.Contents.length); 
});
Ankur
  • 496
  • 1
  • 6
  • 11
Mohamed Najiullah
  • 403
  • 5
  • 12
  • let me tell what I am trying to build. I am building dashboad kind of thing which will display the count of all objects available in a bucket(it can have million objects of small size). I don't actually need any other details for the objects. That is the reason I am looking for alternative of listObjects() which will iterate through all the objects and then find the count.Btw, do you have any handy link which depicts diff between listObjects() , listObjectsV2() to understand what is the difference and also how performance is improved etc.plz suggest me the best approach for my requirement. – user12 Nov 13 '17 at 04:08
  • @Swaroop I found the following [here](https://issues.apache.org/jira/browse/HADOOP-13421) _Unlike version 1 of the S3 List Objects API, version 2 by default does not fetch object owner information, which S3A doesn't need anyway. By switching to v2, there will be less data to transfer/process. Also, it should be more robust when listing a versioned bucket with "a large number of delete markers"_ – Mohamed Najiullah Nov 13 '17 at 05:58