1

I'm on the newest AWS CLI version - 2.7.24

Trying to list all the files from all the folders I have (with a certain extension) on this path:

s3://myfiles/folders/

"folders" have this structure:

folder1
 - item
 - item
folder2
 - item
 - item
folder3
 - item
 - item

My aws cli command is:

aws s3 ls --recursive s3://myfiles/folders/ -> Which works fine. But when I add --include, it doesn't work. Error: unknown options

Example: aws s3 ls --recursive --exclude * --include "*.txt" s3://myfiles/folders/

Error: Unknown options: --exclude, , --include,*.txt

I did pip install -U awscli

I tried a lot of internet and stackoverflow stuff but nothing worked.

Any ideas?

Milos
  • 23
  • 1
  • 6
  • Look at what `aws s3 ls help` tells you – Paolo Aug 17 '22 at 14:19
  • @Paolo it doesn't even have any options about --include or --exclude. Is it possible that it's been evicted from aws s3 ls? – Milos Aug 17 '22 at 14:24
  • 1
    Those options are valid for `aws s3 cp`, not for `aws s3 ls` – Paolo Aug 17 '22 at 14:29
  • I think your `*` and maybe `"*.txt"` are being expanded by the shell before being passed into the AWS cli command. You can check this by running `echo aws s3 ls --recursive --exclude * --include "*.txt" s3://myfiles/folders/` to see what is getting expanded. At a minimum you need to wrap the `*` in quotes. – Mark B Aug 17 '22 at 14:36
  • @MarkB as a result I get this `aws s3 ls --recursive --exclude * --include *.txt s3://myfiles/folders/` And yes, I wrapped `*` with quotes – Milos Aug 17 '22 at 14:40
  • `aws s3 ls` never supported `--include` or `--exclude`. There is an [open feature request](https://github.com/aws/aws-cli/issues/4832) for such support, but as it is now, you'll need to use some external program to accomplish this goal. – Anon Coward Aug 17 '22 at 15:13
  • @AnonCoward any ideas on how to do it? – Milos Aug 17 '22 at 15:27
  • You can use something like 'grep', see my answer below – Paolo Aug 17 '22 at 16:10

3 Answers3

3

You can easily build the desired functionality by piping the result of the aws cli command to grep or similar.

In Bash:

Here's an example which mirrors the "include" functionality:

$ aws s3 ls s3://bucket --recursive | grep ".*.txt.*"

Here's an example which mirrors the "exclude" functionality:

$ aws s3 ls s3://bucket --recursive | grep -v ".*.txt.*"

In Powershell:

Here's an example which mirrors the "include" functionality:

$ aws s3 ls s3://bucket --recursive | Select-String ".*.txt.*"

Here's an example which mirrors the "exclude" functionality:

$ aws s3 ls s3://bucket --recursive | Select-String ".*.txt.*" -NotMatch
Paolo
  • 21,270
  • 6
  • 38
  • 69
1

Currently there is no such option as

aws s3 ls --include

or

aws s3 ls --exclude

There is an open feature request for that, but no solution so far.

Logemann
  • 2,767
  • 33
  • 53
Milos
  • 23
  • 1
  • 6
0

You can do expansions with s3 ls if you want to match the beginning of the path...

If I want to find all paths that start with a capital "C" inside a certain directory I can do this:

aws s3 ls my-bucket/my-folder/C

And it'll match all folders that start with "C" inside my-bucket/my-folder/.

cd3k
  • 719
  • 5
  • 8