Stack Overflow
Stack Overflow

Questions tagged [amazon-s3]

Amazon S3 (simple storage service) is an online object storage service from Amazon Web Services. QUESTIONS MUST BE ABOUT PROGRAMMING. Questions about general S3 support, functionality, configuration, etc. are OFF-TOPIC.

Amazon S3 (Simple Storage Service) is an online strongly consistent storage web service offered by Amazon Web Services. Amazon S3 provides storage through a simple web services interface. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites.

References

Tutorials

49879 questions
235
votes
8 answers

Open S3 object as a string with Boto3

I'm aware that with Boto 2 it's possible to open an S3 object as a string with: get_contents_as_string() Is there an equivalent function in boto3 ?
Gahl Levy
  • 3,211
  • 2
  • 13
  • 7
232
votes
34 answers

How can I tell how many objects I've stored in an S3 bucket?

Unless I'm missing something, it seems that none of the APIs I've looked at will tell you how many objects are in an /. Is there any way to get a count?
fields
  • 4,433
  • 4
  • 27
  • 32
227
votes
3 answers

How to rename AWS S3 Bucket

After all the tough work of migration etc, I just realised that I need to serve the content using CNAME (e.g media.abc.com). The bucket name needs to start with media.abc.com/S3/amazon.com to ensure it works perfectly. I just realised that S3…
Carson Lee
  • 2,673
  • 3
  • 20
  • 23
220
votes
7 answers

How to make all Objects in AWS S3 bucket public by default?

I am using a PHP library to upload a file to my bucket. I have set the ACL to public-read-write and it works fine but the file is still private. I found that if I change the Grantee to Everyone it makes the file public. What I want to know is how do…
condo1234
  • 3,285
  • 6
  • 25
  • 34
216
votes
4 answers

How can I use wildcards to `cp` a group of files with the AWS CLI?

I'm having trouble using * in the AWS CLI to select a subset of files from a certain bucket. Adding * to the path like this does not seem to work: aws s3 cp s3://data/2016-08* .
metersk
  • 11,803
  • 21
  • 63
  • 100
216
votes
8 answers

Downloading folders from aws s3, cp or sync?

If I want to download all the contents of a directory on S3 to my local PC, which command should I use cp or sync ? Any help would be highly appreciated. For example, if I want to download all the contents of "this folder" to my desktop, would it…
BFlint
  • 2,407
  • 2
  • 16
  • 20
213
votes
7 answers

How to specify credentials when connecting to boto3 S3?

On boto I used to specify my credentials when connecting to S3 in such a way: import boto from boto.s3.connection import Key, S3Connection S3 = S3Connection( settings.AWS_SERVER_PUBLIC_KEY, settings.AWS_SERVER_SECRET_KEY ) I could then use S3 to…
Robert Brax
  • 6,508
  • 12
  • 40
  • 69
212
votes
22 answers

How do you search an amazon s3 bucket?

I have a bucket with thousands of files in it. How can I search the bucket?
vinhboy
  • 8,542
  • 7
  • 34
  • 44
206
votes
15 answers

Save Dataframe to csv directly to s3 Python

I have a pandas DataFrame that I want to upload to a new CSV file. The problem is that I don't want to save the file locally before transferring it to s3. Is there any method like to_csv for writing the dataframe to s3 directly? I am using…
user2494275
  • 2,063
  • 2
  • 13
  • 4
199
votes
8 answers

How to write a file or data to an S3 object using boto3

In boto 2, you can write to an S3 object using these methods: Key.set_contents_from_string() Key.set_contents_from_file() Key.set_contents_from_filename() Key.set_contents_from_stream() Is there a boto 3 equivalent? What is the boto3 method for…
jkdev
  • 11,360
  • 15
  • 54
  • 77
199
votes
22 answers

Getting Access Denied when calling the PutObject operation with bucket-level permission

I followed the example on http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_examples.html#iam-policy-example-s3 for how to grant a user access to just one bucket. I then tested the config using the W3 Total Cache Wordpress plugin. The…
Greg
  • 12,119
  • 5
  • 32
  • 34
198
votes
10 answers

AWS Error Message: A conflicting conditional operation is currently in progress against this resource

I'm getting this error intermittently. I have a program that uses the java aws sdk and loads over the 10s of thousands of small files to s3. I see this error intermittently. Could not find any helpful answer after doing a quick search on the…
user1172468
  • 5,306
  • 6
  • 35
  • 62
197
votes
6 answers

Serving gzipped CSS and JavaScript from Amazon CloudFront via S3

I've been looking for ways of making my site load faster and one way that I'd like to explore is making greater use of Cloudfront. Because Cloudfront was originally not designed as a custom-origin CDN and because it didn't support gzipping, I have…
Donald Jenkins
  • 3,485
  • 8
  • 33
  • 35
192
votes
31 answers

Quick way to list all files in Amazon S3 bucket?

I have an amazon s3 bucket that has tens of thousands of filenames in it. What's the easiest way to get a text file that lists all the filenames in the bucket?
Steve
  • 4,033
  • 5
  • 32
  • 29
187
votes
10 answers

Amazon S3 direct file upload from client browser - private key disclosure

I'm implementing a direct file upload from client machine to Amazon S3 via REST API using only JavaScript, without any server-side code. All works fine but one thing is worrying me... When I send a request to Amazon S3 REST API, I need to sign the…
Olegas
  • 10,349
  • 8
  • 51
  • 72
1
2
3
…
99 100