Delete files from folder in S3 bucket

Question

I have an AWS S3 bucket test-bucket with a data folder. The data folder will have multiple files.

I am able to delete the files in the S3 bucket. But what I want is to delete the files in the data folder without deleting the folder.

I tried the following:

aws s3 rm  s3://test-bucket/data/*

Also checked using --recursive option, but that does not work.

Is there a way I can delete the files in the folder using AWS CLI?

There is no such thing as a folder in the current S3 API. It's simply a flat namespace. If you happen to have keys named like "data/foo" and "data/bar", you can emulate a folder structure with S3's [delimited list](http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html) API. Some utilities add a 0-byte file at the base "folder name" like "data/$FOLDER$" or "data/" in order to preserve the appearance of a folder even when empty. But if your S3 client isn't aware of that special file's meaning as a folder placeholder, it'll just show it as any 0-byte object. — perpetual_check, Sep 15 '17 at 20:39
Remove the `*`. The shell will interpolate it into something you don't want. Add `--recursive`. See what that does. — Michael - sqlbot, Sep 15 '17 at 22:47
Don't get hung-up no folders, whether they are there or not. It doesn't matter whether they are there or not. — John Rotenstein, Sep 16 '17 at 01:11
@sethwm So I need to set delimiter="/" and prefix="test-bucket/data/" in aws cli right? — cooldev, Sep 16 '17 at 02:49

score 10 · Accepted Answer · edited Sep 16 '17 at 11:43

10

Following aws cli command worked:

aws s3 rm s3://test-bucket --recursive --exclude="*" --include="data/*.*"

edited Sep 16 '17 at 11:43

Michael - sqlbot

169,571
25
353
427

answered Sep 16 '17 at 03:17

cooldev

497
1
3
17

1

This also deleted the folder for me. In the case s3://folder1/folder2/folder3/hello.txt, `aws s3 rm s3://folder1/folder2/folder3 --exclude="*" --include="*.txt"` or `aws s3 rm s3://folder1/folder2 --exclude="*" --include="folder3/*.txt"` both removed folder3. – Jesse Zhuang Dec 10 '19 at 19:42
2

@JesseZhuang Folders don't exist on S3 they are just made up from the file names, so once you delete the last file in a folder, the folder will go too. – Malvineous Feb 08 '21 at 06:11

score 2 · Answer 2 · answered Nov 17 '20 at 08:44

2

aws s3 rm s3://bucket/folder1/folder2/ --recursive --dryrun

From what I see happening when I try it, adding the slash at the end means delete below folder2, not including folder2.

answered Nov 17 '20 at 08:44

Nick Weavers

534
7
23

This solution works fine when used it inside Python code but the one that was selected as the solution did its job but it did not return and Python code looked like hang. – Ahmad Dec 16 '22 at 21:21

score 1 · Answer 3 · answered Mar 17 '19 at 09:29

you can do it using aws cli : https://aws.amazon.com/cli/ and some unix command.

this aws cli commands should work:

aws s3 rm s3://<your_bucket_name> --exclude "*" --include "<your_regex>"

if you want to include sub-folders you should add the flag --recursive

or with unix commands:

aws s3 ls s3://<your_bucket_name>/ | awk '{print $4}' | xargs -I%  <your_os_shell>   -c 'aws s3 rm s3:// <your_bucket_name>/% $1'

explanation: list all files on the bucket --pipe--> get the 4th parameter(its the file name) --pipe--> run delete script with aws cli

score 0 · Answer 4 · answered Jan 06 '23 at 07:53

We can remove all files including sub-folders as well from AWS-S3-BUCKET In Node.js by using the below function.

the same command can be used in AWS-CLI by configuring AWS in the command-line.

function removeAllFilesFromBucket(){
    const S3_REGION = "eu-west-1";
    const S3_BUCKET_NAME = "sample-staging";
    let filePathBucket = S3_BUCKET_NAME+'/assets/videos';

    let awsS3ShellCommand = 'aws s3 rm s3://'+filePathBucket+' --region '+S3_REGION+' --recursive';                 
    var { exec } = require('child_process');  
    exec(awsS3ShellCommand, (err, stdout, stderr) => {
        if (err) {
            console.log('err',err);                        
            return;
        }else{
            console.log('Bucket files and sub-folders Deleted successfully !!!'); 
            console.log(`stdout: ${stdout}`);
            console.log(`stderr: ${stderr}`); 
        }
    });  
}

Delete files from folder in S3 bucket

4 Answers4

Linked