Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [amazon-s3]

This is for questions on Amazon's cloud storage service.

1120 questions
18
votes
2 answers

Deleting S3 files with a given prefix only

We have a bucket with more than 500,000 objects in it. I'm assigned a job where I've to delete files which have a specific prefix. There are around 300,000 files with the given prefix in the bucket. For eg If there are 3…
Axel
  • 421
  • 2
  • 8
  • 18
16
votes
4 answers

Recursively changing the content-type for files of a given extension on Amazon S3

I have a large S3 bucket with a nested "folder" structure containing (among other things) static .json and .md files. Theses files are being served by S3 as text/plain rather than the correct application/json and text/markdown. I have updated the…
Gabriel Bauman
  • 313
  • 1
  • 2
  • 10
16
votes
3 answers

Backing up data stored on Amazon S3

I have an EC2 instance running a web server that stores users' uploaded files to S3. The files are written once and never change, but are retrieved occasionally by the users. We will likely accumulate somewhere around 200-500GB of data per year. We…
Fiver
  • 213
  • 2
  • 8
15
votes
4 answers

Amazon S3 Console: How to find total number of files with in a folder?

So I can see S3 console and can get into my folder with files listing but I don't find way to find total number of files in it other than going thru pagination which does not work considering thousands of files.
Volatil3
  • 275
  • 1
  • 3
  • 11
15
votes
3 answers

How to config Amazon Route53 working without www in sub-domain

edit: Amazon now supports this. http://aws.typepad.com/aws/2012/12/root-domain-website-hosting-for-amazon-s3.html I have my domain config in Route53 at Amazon AWS exec ping command in my domain without www $ ping mydomain.com.br ping: cannot…
romuloigor
  • 185
  • 1
  • 1
  • 7
15
votes
5 answers

How to make s3fs work with IAM roles?

We are using IAM roles to maintain the machines. We now planning to mount the s3 to our ec2 instances and do the processing as per our need. We been using s3fs tool for mounting. But that doesn't seems to be working with the IAM roles. Are there any…
sriram
  • 277
  • 2
  • 3
  • 9
15
votes
5 answers

Software for using Amazon S3 as backup service

What software do you recommend for using Amazon S3 as a placeholder for daily backups? I need it to be fully automated, and easy to restore data.
Frode Lillerud
  • 1,656
  • 3
  • 18
  • 20
14
votes
3 answers

Access Amazon S3 Static Website Through IPv6

I host a static website on Amazon S3 with DNS hosted on Amazon Route 53. I'd like my site to be accessible via IPv4 and IPv6 (because it's about time). When I use the Route 53 AWS Management Console to add an A (IPv4) "Alias" record for my S3…
user101340
14
votes
3 answers

Wildcard Subdomain (*.example.com) with Amazon Route53 DNS and S3

I am hosting a static website using Amazon Route53 for DNS and S3 for html files. It is working fine. Current Setup is: example.com. A ALIAS s3-website-us-east-1.amazonaws.com. www.example.com. CNAME…
Jasper
  • 297
  • 2
  • 4
  • 11
14
votes
5 answers

Accessing Amazon S3 from a private VPC subnet

If I have a VPC running and some servers located in the private portion of that network that do backend processing by downloading files from amazon s3, can I access S3 internally to get at those files? Or do I have to access the public internet via…
The Internet
  • 503
  • 2
  • 8
  • 19
14
votes
3 answers

Why is uploading to S3 so slow?

I am using s3cmd to upload to S3: # s3cmd put 1gb.bin s3://my-bucket/1gb.bin 1gb.bin -> s3://my-bucket/1gb.bin [1 of 1] 366706688 of 1073741824 34% in 371s 963.22 kB/s I am uploading from Linode, which has an outgoing bandwidth cap of 50…
Tom Marthenal
  • 2,116
  • 7
  • 25
  • 37
14
votes
3 answers

Forcing CloudFront to pass-through the latest HTML file from S3

Background I'm hosting a static site on S3, with CloudFront over the top. The issue I have is with my HTML files. According to CloudFront's FAQ: Amazon CloudFront uses these cache control headers to determine how frequently it needs to check the…
isNaN1247
  • 1,675
  • 3
  • 15
  • 20
14
votes
10 answers

S3-compatible Object Storage systems

Has anyone come across a near 100% Amazon S3 RESTful API compatible object storage system? What I am after is a layer that sit on top of any (preferably a POSIX) filesystem that provides Amazon S3 style RESTful API to store (PUT), retrieve (GET),…
CodeMedic
  • 304
  • 1
  • 2
  • 11
14
votes
7 answers

What is a practical way to mirror an Amazon S3 bucket?

I want to mirror my Amazon S3 buckets. I want to do this because 1) I don't want all my data only existing with one provider; and 2) in case of software error or security breach I want to have the data backed up. I can mirror to a local disk with…
Ben
  • 173
  • 1
  • 2
  • 7
13
votes
1 answer

What does new "S3 Increased Request Rate Performance" annoucement mean

On 17 July 2018 there was an official AWS announcement explaining that there is no longer any need to randomize the first characters of every S3 object key to achieve maximum performance:…
John Rees
  • 233
  • 2
  • 5
1 2
3
…
74 75