Stack Overflow
Stack Overflow

Questions tagged [amazon-s3]

Amazon S3 (simple storage service) is an online object storage service from Amazon Web Services. QUESTIONS MUST BE ABOUT PROGRAMMING. Questions about general S3 support, functionality, configuration, etc. are OFF-TOPIC.

Amazon S3 (Simple Storage Service) is an online strongly consistent storage web service offered by Amazon Web Services. Amazon S3 provides storage through a simple web services interface. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites.

References

Tutorials

49879 questions
142
votes
8 answers

Amazon S3 boto - how to delete folder?

I created a folder in s3 named "test" and I pushed "test_1.jpg", "test_2.jpg" into "test". How can I use boto to delete folder "test"?
wade huang
  • 1,435
  • 2
  • 11
  • 7
141
votes
34 answers

The AWS Access Key Id does not exist in our records

I created a new Access Key and configured that in the AWS CLI with aws configure. It created the .ini file in ~/.aws/config. When I run aws s3 ls it gives: A client error (InvalidAccessKeyId) occurred when calling the ListBuckets operation: The AWS…
kaushikdr
  • 1,699
  • 2
  • 12
  • 18
140
votes
5 answers

Get last modified object from S3 using AWS CLI

I have a use case where I programmatically bring up an EC2 instance, copy an executable file from S3, run it and shut down the instance (done in user-data). I need to get only the last added file from S3. Is there a way to get the last modified file…
wishy
  • 1,748
  • 2
  • 12
  • 14
138
votes
14 answers

How to get response from S3 getObject in Node.js?

In a Node.js project I am attempting to get data back from S3. When I use getSignedURL, everything works: aws.getSignedUrl('getObject', params, function(err, url){ console.log(url); }); My params are: var params = { Bucket:…
Sara Fuerst
  • 5,688
  • 8
  • 43
  • 86
133
votes
34 answers

AWS CLI S3 A client error (403) occurred when calling the HeadObject operation: Forbidden

I'm trying to setup a Amazon Linux AMI(ami-f0091d91) and have a script that runs a copy command to copy from a S3 bucket. aws --debug s3 cp s3://aws-codedeploy-us-west-2/latest/codedeploy-agent.noarch.rpm . This script works perfectly on my local…
MojoJojo
  • 3,897
  • 4
  • 28
  • 54
133
votes
5 answers

How to use AWS S3 CLI to dump files to stdout in BASH?

I'm starting a bash script which will take a path in S3 (as specified to the ls command) and dump the contents of all of the file objects to stdout. Essentially I'd like to replicate cat /path/to/files/* except for S3, e.g. s3cat…
Neil C. Obremski
  • 18,696
  • 24
  • 83
  • 112
131
votes
16 answers

How to check if a specified key exists in a given S3 bucket using Java

I would like to check if a key exists in a given bucket using Java. I looked at the API but there aren't any methods that are useful. I tried to use getObject but it threw an exception.
in His Steps
  • 3,075
  • 6
  • 30
  • 38
131
votes
10 answers

How to import a text file on AWS S3 into pandas without writing to disk

I have a text file saved on S3 which is a tab delimited table. I want to load it into pandas but cannot save it first because I am running on a heroku server. Here is what I have so far. import io import boto3 import os import pandas as…
alpalalpal
  • 1,583
  • 2
  • 12
  • 14
130
votes
10 answers

Amazon S3 upload file and get URL

Is it possible to upload a txt/pdf/png file to Amazon S3 in a single action, and get the uploaded file URL as the response? If so, is AWS Java SDK the right library that I need to add in my java struts2 web application? Please suggest me a solution…
Sangram Anand
  • 10,526
  • 23
  • 70
  • 103
128
votes
8 answers

S3 - What Exactly Is A Prefix? And what Ratelimits apply?

I was wondering if anyone knew what exactly an s3 prefix was and how it interacts with amazon's published s3 rate limits: Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500…
dm03514
  • 54,664
  • 18
  • 108
  • 145
127
votes
4 answers

Error "Read-only file system" in AWS Lambda when downloading a file from S3

I'm seeing the below error from my lambda function when I drop a file.csv into an S3 bucket. The file is not large and I even added a 60 second sleep prior to opening the file for reading, but for some reason the file has the extra ".6CEdFe7C"…
user1530318
  • 25,507
  • 15
  • 37
  • 48
127
votes
18 answers

Boto3 to download all files from a S3 Bucket

I'm using boto3 to get files from s3 bucket. I need a similar functionality like aws s3 sync My current code is #!/usr/bin/python import boto3 s3=boto3.client('s3') list=s3.list_objects(Bucket='my_bucket_name')['Contents'] for key in list: …
Shan
  • 2,141
  • 2
  • 17
  • 32
125
votes
3 answers

How to get more than 1000 objects from S3 by using list_objects_v2?

I have more than 500,000 objects on s3. I am trying to get the size of each object. I am using the following python code for that import boto3 bucket = 'bucket' prefix = 'prefix' contents = boto3.client('s3').list_objects_v2(Bucket=bucket, …
tahir siddiqui
  • 1,371
  • 2
  • 8
  • 8
125
votes
16 answers

AWS S3 display file inline instead of force download

For some reason files in my S3 bucket are being forced as downloads instead of displaying in-line so if I copy an image link and paste it into address bar and then navigate to it, it will promote my browser to download it. Instead I actually have to…
Cl'
  • 1,535
  • 2
  • 10
  • 17
123
votes
11 answers

Can Apache Spark run without Hadoop?

Are there any dependencies between Spark and Hadoop? If not, are there any features I'll miss when I run Spark without Hadoop?
tourist
  • 4,165
  • 6
  • 25
  • 47
1 2 3
…
99 100