23

I have a bucket in S3 (Infrequent access) containing 2 billion objects. It is too big to delete in the console or over the api without taking years.

I can create a lifecycle rule to expire and delete the objects but the calculator predicts this will cost me >$20,000. Is that correct? Is there a better way to delete a bucket?

I have a file effectively containing a list of all the objects in that bucket if that helps.

Update 2021:

An answer below from @MAP points out that there is now an "Empty" button. I haven't tested yet, but looks like the way to go (I'll accept that answer once tested):

screenshot of empty button

matt burns
  • 24,742
  • 13
  • 105
  • 107
  • That seems is incorrect. What *exactly* is the calculator showing? – Michael - sqlbot Jan 19 '19 at 04:31
  • I was partly misreading the calculator, I assumed DELETES would be in the GET requests or the other requests, both costs thousands. – matt burns Jan 23 '19 at 16:16
  • 2B lifecycle transitions for S3 IA still cost $20k though – matt burns Jan 23 '19 at 16:17
  • The "Empty" button makes some `LIST` requests to S3 to retrieve the objects to remove. So, the `DELETE` operations are free but not the `LIST` ones. – riccardo.cardin May 30 '22 at 13:42
  • Were the 2 billion IA objects still within the minimum billing duration for their storage class? If so early deletion fees will be charged for the remaining billable duration. – vdm Oct 19 '22 at 12:25

4 Answers4

17

If you have a list of all the objects available then you can certainly use Multi Delete Object action. Apparently this API is free. I would create AWS Step Functions state machine to loop through the file and delete 1000 objects at a time. 1000 appears to be the limit.

It will take around 2M step function transactions to delete all the objects in the bucket. As per the pricing for step function it will cost you around $50 + cost of Lambda invocations around $1 so total cost roughly $51.

Update

Using Lambda or Step Functions is probably not the most cost effective option because both ways you will need to read the file (that contains object keys) from some source such as S3. So I think running the script from local machine or any EC2 linux screen appears to be the best option.

A.Khan
  • 3,826
  • 21
  • 25
  • Thanks, this is the way we are are doing it. – matt burns Jan 23 '19 at 16:14
  • 1
    well, kinda, we're using Flink to rattle through it, but the principle is the same. Thanks for the tip about the multi delete api :) – matt burns Jan 23 '19 at 16:18
  • 20
    Big warning for anyone doing this: We only realised afterwards that we have versioning enabled for this bucket... urgh. Multi object delete simply added "delete markers" to each object. Total size just went up and number of object doubled! Have disabled versioning and running again... – matt burns Jan 27 '19 at 21:22
  • Rather than disable versioning on the bucket, you can reduce the number of days old versions are kept for in the lifecycle policy. – vdm Oct 19 '22 at 12:11
  • This has been marked as the "best" answer. It is the least correct answer. – vdm Oct 19 '22 at 12:21
14

In 2021, anyone who comes across this question may benefit to know that AWS console now provides an empty button.

Select the bucket and click on "empty" button and all objects versioned or not versioned would be emptied/deleted. Depending on the number of objects it can take minutes to days.

Jens
  • 69,818
  • 15
  • 125
  • 179
MAP
  • 141
  • 1
  • 2
  • 1
    In browser dev tools or Cloudtrail, you can see that the Empty button is implemented by the console listing prefixes and sending DeleteObjects requests for the results. The ListObjectsV2 requests are not free. Also if those objects are being deleted early from non-Standard storage class, early deletion fees will be charged. – vdm Oct 19 '22 at 12:19
5

Expiration lifecycle rules are free. From the original feature announcement:

As with standard delete requests, Amazon S3 doesn’t charge you for using Object Expiration.

mR_fr0g
  • 8,462
  • 7
  • 39
  • 54
PencilBow
  • 979
  • 11
  • 17
  • 1
    The delete requests and policies are free, but would potentially result in early deletion fees for storage classes other than Standard, if objects are deleted before the minimum billable storage duration. – vdm Oct 19 '22 at 12:18
3

Delete operations are for free. You can create a lifecycle Policy to automate a bulk delete.

I would start with a small number of objects first and check billing report to 100% confirm that the delete will not be charged, then go for the rest.

Sébastien Stormacq
  • 14,301
  • 5
  • 41
  • 64
  • 3
    Delete ops are free, but lifecycle transitions aren't as far as I know. Calculator predicts 20k – matt burns Jan 23 '19 at 16:17
  • Are delete operations free for IA? That was my assumption, and for other classes they are, but the [pricing page](https://aws.amazon.com/s3/pricing/) implies that for IA and 1ZIA they are not actually free. – Michael - sqlbot Jan 23 '19 at 16:53
  • I confirm the delete is for free, even on IA. I am checking with the service team the point Matt made above about Lifecycle transitions. – Sébastien Stormacq Jan 23 '19 at 17:12
  • 2
    @SébastienStormacq Did you ever receive a response from the service team about the price for using lifecycle transitions to delete? – Jarett Millard Apr 06 '21 at 15:30
  • 1
    The DeleteObject request and lifecycle policy is free. BUT Infrequent Access (IA) storage classes have minimum billable storage durations. If you delete before this minimum duration, you are charged the rest of the duration as an "early deletion fee". – vdm Oct 19 '22 at 12:08