25

My Amazon S3 bucket has a folder structure that looks like the below.

  • bucket-name\00001\file1.txt
  • bucket-name\00001\file2.jpg
  • bucket-name\00002\file3.doc
  • bucket-name\00001\file4.ppt

If I only know file name file3.doc and bucket name bucket-name how can i search for file3.doc in bucket-name. If I knew, it is in folder 00002, I could simply go to the folder and start typing the file name but I have no way to know in which folder the file I am searching for is under.

Fred Rogers
  • 413
  • 2
  • 8
  • 19
  • 1
    Unfortunately I think your options are limited. Recursive loop will probably be your best best. S3 isn't supposed to be used in this manner – Matthew Merryfull Oct 01 '15 at 06:01
  • Do you wish to obtain it programmatically, or is a command-line sufficient? In what context are you looking for the file (eg is it a once-off request, regular job, part of a bigger app, etc)? – John Rotenstein Oct 02 '15 at 23:30

3 Answers3

34

You can easily do this with the AWS CLI.

aws s3 ls s3://BUCKET-NAME/ --recursive | grep FILE-NAME.TXT
hfranco
  • 943
  • 4
  • 16
  • 26
14

Using only the AWS CLI, you can run a list-objects against the bucket with the --query parameter. This will not be a fast operation, as it runs locally after fetching the file list, rather than inside s3's api.

$ aws s3api list-objects --bucket bucket-name --query "Contents[?contains(Key, 'file3')]"

[
    {
        "LastModified": "2017-05-31T20:36:28.000Z",
        "ETag": "\"b861daa5cc3775f38519f5de6566cbe7\"",
        "StorageClass": "STANDARD",
        "Key": "00002/file3.doc",
        "Owner": {
            "DisplayName": "owner",
            "ID": "123"
        },
        "Size": 27032
    }
]

The benefit of using --query over just piping to grep is that you'll get the full response including all available metadata usually included in list-objects, without having to monkey around with before and after arguments for the grep.

See this post on Finding Files in S3 for further information including a similar example which shows the benefit of having metadata, when files of the same name end up in different directories.

Luke Waite
  • 2,265
  • 1
  • 25
  • 23
3

You'll probably need to use a command line tool like s3cmd if you don't know where it is at all:

s3cmd --recursive ls s3://mybucket | grep "file3"

but some limited search is possible:

https://stackoverflow.com/a/21836343/562557

Community
  • 1
  • 1
thebenedict
  • 2,539
  • 3
  • 20
  • 29
  • 5
    These days, it's normally recommended to use the [AWS Command-Line Interface (CLI)](http://aws.amazon.com/cli/), which has functionality for all AWS services (not just Amazon S3). – John Rotenstein Oct 02 '15 at 23:31