37

I was thinking about deleting and then re-creating bucket (bad option which I realised later).

Then how can delete all objects from the bucket?

I tried this : http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Bucket.delete_objects

But it deletes multiple objects not all.

can you suggest what is the best way to empty bucket ?

Tushar Niras
  • 3,654
  • 2
  • 22
  • 24
  • This is a limitation of the S3 API - [delete multiple object](http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html). You will have to implement pagination and delete 1000 at a time. – AChampion Apr 10 '17 at 17:51

4 Answers4

77

Just use aws cli.

aws s3 rm s3://mybucket --recursive

Well, for longer answer if you insists to use boto3. This will send a delete marker to s3. No folder handling required. bucket.Object.all will create a iterator that not limit to 1K .

import boto3    
s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket')
# suggested by Jordon Philips 
bucket.objects.all().delete()
mootmoot
  • 12,845
  • 5
  • 47
  • 44
  • 12
    You can also just do `bucket.objects.all().delete()`, though it gets a bit more complicated if your bucket is versioned – Jordon Phillips Apr 11 '17 at 21:15
  • @JordonPhillips Thank for your answer. This worked for me as I'm not using versioning for my bucket. But will this find all the objects inside the bucket and then delete them one by one? what happens when there are huge amounts of data in bucket ? Will it be slow? – Tushar Niras Apr 12 '17 at 05:32
  • @TusharNiras : whether object store or normal file system, `delete` is just about removing the index. – mootmoot Apr 12 '17 at 08:04
  • 2
    @TusharNiras, Under the hood boto3 will use [`delete_objects`](http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.delete_objects) when you use that snippet, so you will make one request per page of objects (up to 1000 at a time). This is as fast as you can make it without threading. – Jordon Phillips Apr 12 '17 at 15:27
  • can you demonstrate how to generate a client without the configuration file? like https://boto3.readthedocs.io/en/latest/guide/configuration.html#method-parameters – Harry Moreno Apr 24 '17 at 21:06
  • @HarryMoreno : please post as a new question. – mootmoot Apr 25 '17 at 07:25
  • wrote it up as a blogpost http://harrymoreno.com/2017/04/24/How-to-fill-and-empty-an-s3-bucket-with-python.html – Harry Moreno Apr 25 '17 at 21:16
  • @HarryMoreno : Do you mean AWS IAM Assume roles and STS ? – mootmoot Apr 26 '17 at 07:39
  • If you don't have the DeleteObject permission, the objects will not be deleted and an error will NOT be raised. After I added the permission the objects were deleted as expected. – sb89 Dec 25 '17 at 12:08
  • Thanks very helpful. The OP just asks to "empty the bucket" but you've also added `bucket.delete()` so I've edited it to add a comment to that effect if that's alright. – Davos Jan 02 '19 at 05:04
  • This literally does nothing for me. It returns without error, but the bucket I specify continues to exist. – Cerin May 14 '19 at 19:33
  • 2
    @Davos After second though, I have remove bucket.delete() , just in case people testing the script get their bucket deleted. – mootmoot May 15 '19 at 09:56
  • @Cerin : Can you check your bucket policy and your user rights? – mootmoot May 15 '19 at 09:56
  • 9
    Looks like you can now skip the `.all()` and just do `bucket.objects.delete()`. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Bucket.objects. – Ryan Feb 17 '20 at 22:31
26

If versioning is enabled, there's a similar call to the other answer to delete all object versions:

import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket('bucket-name')
bucket.object_versions.delete()
kgutwin
  • 867
  • 1
  • 10
  • 15
  • 1
    i found this to be incredibly slow. It seems to collect and return details about each object when complete, maybe this is why? any hints there? (I know the question asked specifically for boto3, but the console "empty" option is much faster) – Peter McIntyre Jan 21 '22 at 02:28
17

Based on previous responses, and adding check of versioning enabled, you can empty a bucket, with versions enabled or not doing:

s3 = boto3.resource('s3')
s3_bucket = s3.Bucket(bucket_name)
bucket_versioning = s3.BucketVersioning(bucket_name)
if bucket_versioning.status == 'Enabled':
    s3_bucket.object_versions.delete()
else:
    s3_bucket.objects.all().delete()
Gonzalo Odiard
  • 1,238
  • 12
  • 19
0

Or if you need to use low-level operations on boto3 and yoy need to use boto3.client instead of boto3.resource here's a code that can help. In which I called all the objects with s3_client.list_objects_v2 and extracted the key on them in order to delete them with s3_client.delete_objects

import boto3

s3_client = boto3.client("s3")
objects = s3_client.list_objects_v2(Bucket = "bucket-name")["Contents"]
objects = list(map(lambda x: {"Key":x["Key"]},objects))
s3_client.delete_objects(Bucket = "bucket-name", Delete = {"Objects":objects})
JULIO SIMON PEDRO BOHRQUEZ HUR
  • 21
  • 4