Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [amazon-s3]

This is for questions on Amazon's cloud storage service.

1120 questions
10
votes
2 answers

How can I move s3 objects >5gb from one bucket to another?

Objects larger than 5GB are supported with multi-part uploading. We have 5gb+ files in one bucket already that I'd like to move to another, under the same AWS account. When I issue the command using s3cmd: s3cmd mv s3://BUCKET1/OBJECT1…
Ben Keating
  • 211
  • 1
  • 2
  • 5
10
votes
6 answers

Does a CDN still work even when my server is down?

I am a website owner planning to use Amazon's cloudfront S3. I have been reading all the stuffs about what a CDN can do but i still have an unanswered question. Does a CDN still work even when my main server is down. That's the main reason i am…
Tola
  • 101
  • 1
  • 3
10
votes
8 answers

S3 Interfaces for Backup, etc

Since none of the S3 "file systems" are compatible right now, what s3 gateway tool do you recommend for mounting a remote S3 bucket for backups? I've been using S3FS, but I've heard a lot of people complaining about the poor quality of the code; I…
Mikeage
  • 2,741
  • 6
  • 26
  • 37
10
votes
3 answers

Amazon S3 not sending Content-Type header

I have an application that downloads content from various sources. It relies on the "Content-Type" header being set on images. The majority of web-servers do this correctly but it appears Amazon S3 server is not setting the Content-Type. I assume…
Luke____
9
votes
1 answer

Amazon EKS: how to configure S3 access for worker nodes?

How can I configure an EKS cluster to automatically allow S3 access from worker nodes? I've set up an EKS cluster following the Getting Started guide and have run the example Guest Book app. Now I want to use Snakemake to run bioinformatics…
jackkamm
  • 251
  • 1
  • 2
  • 5
9
votes
1 answer

Is it possible to avoid costs for invalid requests at AWS S3 or cloudfront during a DDoS attack?

There are many tutorials in the internet, that promote cheap hosting of static websites via AWS S3 + AWS Cloudfront (+ Cloudflare). Example of these would be: Host a website using S3, CloudFront and CloudFlare How to host a website on S3? I was…
user1861174
  • 253
  • 1
  • 3
9
votes
2 answers

how to prune old CodePipeline artifacts

Every build artifact is 39MB, and every time I submit a build, it adds another 39MB artifact to the codepipeline S3 bucket. Is there a way to automatically delete old artifacts?
f.khantsis
  • 331
  • 3
  • 6
  • 13
9
votes
3 answers

Likely causes of NTPD dying unexpectedly and solutions

On a web application which uses s3 for physical document storage, we are experiencing issues with NTP continuously dying. This seems to happen roughly once or twice a day. There is very little information provided when this occurs, other than that…
user275940
  • 191
  • 1
  • 4
9
votes
3 answers

s3cmd: Deleting files or folder does not work

s3cmd s3://MYBUCKET/FOLDER does not work. (folder is empty) s3cmd s3://MYBUCKET/FILE does not work. Output of both commands is "File FILE sucessfully deleted", yet the file remains on the bucket. Deleting with the S3 Webinterface works just fine.
user54883
9
votes
5 answers

how to pipe a mysql dump to s3cmd

I want to transfer a mysql dump, compressed, to s3. I tried: mysqldump -u root -ppassword -all-databases | gzip -9 | s3cmd put s3://bucket/sql/databases.sql.gz but then I get: ERROR: Not enough paramters for command 'put' How can I do this (in one…
zensys
  • 319
  • 3
  • 9
  • 19
9
votes
2 answers

AWS Cloudfront multiple origins - how can we fallback to 2nd origin if not found in the 1st

We're trying to set up a Cloudfront distribution where the default origin is S3, but if the content is not found on S3, we want it to fetch it from an EC2 instance. Can we setup these multiple origins: S3 and EC2 and two behaviors, both defaulting…
adoram
  • 91
  • 1
  • 3
9
votes
2 answers

EC2 - Shared Storage - S3FS or EBS?

Building my web service onto EC2 right now and have a single instance behind a load balancer. I will of course cater for multiple instances. My initial idea was to run all the instances a dumb slaves, and use S3 as local storage. For this, I've…
waxical
  • 344
  • 2
  • 5
  • 12
9
votes
9 answers

How to store 3 million records in key value format?

We have to store basic information about 3 million products. Currently the info is one 180 mb CSV which gets updated quarterly. There will be about 30,000 queries per day, but the queries are just a very simple key value store. We only need to…
Phil
  • 105
  • 1
  • 5
9
votes
3 answers

Serving static files with amazon s3 vs nginx

Let say I have an django application Will it be better (performance-wise) to serve the static files with s3 or to setup an nginx server on a ec2 instance?
Julien Grenier
  • 225
  • 3
  • 8
9
votes
4 answers

Deleting an Amazon AWS S3 bucket with many thousands of files/keys

Is there an easy way of deleting an Amazon S3 bucket that has many tens of thousands of objects within it? I've tried s3cmd with --recursive and --force, but it memory-bloats and has been running for days with little effect.
Lance
  • 91
  • 1
  • 1
  • 3
1 2 3
…
74 75