Check if file exists in s3 using ls and wildcard

Question

Seems so simple, but not getting the syntax right. I want to know if a file exists in my s3 bucket using wildcards. Something like

aws s3 ls s3://my-bucket/folder/*myfile*

The goal is to see if a file called 2016_myfile.txt or a file called 2011_myfile.csv exists within this bucket.

If I run the command, it doesn't return anything even though I know this file exists there.

Very inefficient, but why not do something like "aws s3 ls s3://my-bucket/folder/ | grep myfile" ? That is, if you insist on using the CLI. A python script using Boto would be much more efficient, though it's not a single line command. — Gil Adirim, Oct 04 '16 at 17:17
@GilAdirim Actually that would work as well. I was just surprised that the aws-cli didn't provide something like this and thought I was missing something — Nicros, Oct 04 '16 at 17:41
@GilAdirim I've been looking at Boto and I don't see how it supports wildcards either, do you have a pointer you could provide? — Ken Williams, Dec 15 '16 at 19:30

Chrs · Answer 1 · 2020-11-12T11:12:42.817

65

aws s3 ls does not support globs, but sync does and has a dry run mode. So if you do this (from an empty directory) you should get the results you want:

aws s3 sync s3://my-bucket . --exclude "*" --include "folder/*myfile*" --dryrun

It will produce lines like this for matching files:

(dryrun) download s3://my-bucket/folder/myfile.txt to folder/myfile.txt
(dryrun) download s3://my-bucket/folder/_myfile-foo.xml to folder/_myfile-foo.xml

edited Nov 12 '20 at 11:12

answered Feb 25 '20 at 15:26

Chrs

862
6
10

1

I just tried this, and it worked `aws s3 sync --exclude '*' --include '*.DS_Store' s3://mybucket ./` thank you for that post! +1 – Felipe Alvarez Apr 02 '20 at 12:09
2

This is genius :). Thanks. – John Humphreys Jul 23 '20 at 17:18
1

Amazing. Thank you! – gav.newalkar Nov 09 '20 at 18:41
2

This is so much better that top answer - simply because if you use `aws s3 ls` piped to `grep` you are limited to 1000 results in a page so if your bucket has more than 1000 items you'll not get reliable results – Barry Feb 25 '21 at 15:30
3

This isn't a solution if you want the file timestamp or size that you get with `aws s3 ls` – swdev May 28 '21 at 19:56
works perfect - best answer !! always look on *all* answers, and not just on the accepted one – OrenIshShalom Jun 30 '21 at 12:44
You can even use `aws s3 rm` instead of `aws s3 sync` and then it doesn't matter where you run it from :) – Zionsof Dec 20 '22 at 15:39

score 33 · Accepted Answer · answered Oct 04 '16 at 17:54

33

(re-drafted from comment as it appears this answered the question)

I myself tried, and failed to use wildcards in the aws-cli, and according to the docs, this is not currently supported. Simplest (though least efficient) solution would be to use grep:

aws s3 ls s3://my-bucket/folder/ | grep myfile

Alternatively, you could write a short python/other script to do this more efficiently (but not in a single command)

answered Oct 04 '16 at 17:54

Gil Adirim

1,834
2
21
33

This will produce inaccurate results if you have more than 1000 objects in the bucket. – Dave Yarwood Apr 29 '18 at 00:00
4

@DaveYarwood, I don't think that's true. I think you're talking about the _page-size_ limit, which isn't relevant here. – Arel Mar 18 '19 at 17:41
won't this transfert all filenames to your local machine first? – bonobo Jun 21 '23 at 09:42

score 14 · Answer 3 · answered Sep 01 '17 at 12:19

14

S3 doesn't support wildcard listing. You need to list all the files and grep it.

aws s3 ls s3://mybucket/folder --recursive

Above command will give the list of files under your folder, it searches the files inside the folder as well. Just grep your file name

aws s3 ls s3://mybucket/folder --recursive |grep filename

Suppose if you want to find multiple files, create a regular expression of those and grep it.

answered Sep 01 '17 at 12:19

skipper21

2,165
2
16
13

3

This will produce inaccurate results if you have more than 1000 objects in the bucket. – Dave Yarwood Apr 29 '18 at 00:00
1

@DaveYarwood: could you specify why? I know these operations are not nearly as reliable as analogous filesystem ones, but a quick search didn't turn up much for me. – user1071847 Dec 05 '18 at 14:32
IIRC, there is a limit of 1000 on the number of items that `aws s3 ls` will return. – Dave Yarwood Dec 05 '18 at 19:12
3

@DaveYarwood, do you mean there is a *page-size* limit of 1000, because `aws s3 ls` will clearly return more than 1000 values (as I verified just now). I understand the docs to mean that the page limit is **per API call** happening behind the scenes. – Arel Mar 18 '19 at 17:40

score 3 · Answer 4 · answered Dec 17 '19 at 09:56

3

For Windows User:

aws s3 ls s3://my-bucket/folder/ | findstr myfile

answered Dec 17 '19 at 09:56

Harkirat Singh

49
3

Rico · Answer 5 · 2022-03-07T21:49:49.163

1

2022 Update:

You can use this supported with aws s3 with the --dry-run option:

aws s3 sync s3://my-bucket . --include "folder/*myfile*" --exclude "*" --dryrun

Alternatively,

s3cmd also works for me with grep.

s3cmd ls --recursive s3://mybucket/folder/folder2 | grep filename

edited Mar 07 '22 at 21:49

answered Oct 04 '16 at 17:56

Rico

58,485
12
111
141

Check if file exists in s3 using ls and wildcard

5 Answers5

Linked