23

I need to download files recursively from a s3 bucket. The s3 bucket lets anonymous access.

How to list files and download them without providing AWS Access Key using an anonymous user?

My command is:

aws s3 cp s3://anonymous@big-data-benchmark/pavlo/text/tiny/rankings/uservisits uservisit --region us-east --recursive

The aws compains that:

Unable to locate credentials. You can configure credentials by running "aws configure"

Skarab
  • 6,981
  • 13
  • 48
  • 86

2 Answers2

67

You can use no-sign-request option

aws s3 cp s3://anonymous@big-data-benchmark/pavlo/text/tiny/rankings/uservisits uservisit --region us-east --recursive --no-sign-request

Stan
  • 884
  • 7
  • 7
  • 1
    Yep, when using `localstack` , thats the secret sauce. Thanks! – djangofan Apr 15 '18 at 00:11
  • 1
    Which works great not only for localstack. For example a public s3 bucket you can run this locally towards it `aws s3 ls ryft-public-sample-data --no-sign-request` – Piazzolla Jul 09 '21 at 09:37
  • any way to do it with the aws profile or configuration file rather than this command line option? – Eric Roller Feb 11 '22 at 00:13
  • Correct me if I'm wrong but it seems --no-sign-request is no longer supported in signature version 4. Any idea how to get around this? – David Feb 28 '23 at 22:18
3

you probably have to provide an access keys and secret key, even if you're doing anonymous access. don't see an option for anonymous for the AWS cli.

another way to do this, it to hit the http endpoint and grab the files that way.

In your case: http://big-data-benchmark.s3.amazonaws.com

You will get and XML listing all the keys in the bucket. You can extract the keys and issues requests for each. Not the fastest thing out there but it will get the job done.

For example: http://big-data-benchmark.s3.amazonaws.com/pavlo/sequence-snappy/5nodes/crawl/000741_0

for getting the files curl should be enough. for parsing the xml depending on what you like you can go as lo-level as sed and as high-level as a proper language.

hope this helps.

Mircea
  • 10,216
  • 2
  • 30
  • 46