How to upload files from Amazon EC2 server to S3 bucket?

Question

What is the default way to upload files from a EC2 web server to a S3 bucket for content delivery?

score 3 · Answer 1 · answered Dec 12 '11 at 04:57

The answer is rather dependent on the purpose. For instance, if your EC2 instance is running a PHP application, then using the PHP-SDK would be the best route. Some applications might find it useful to mount an S3 bucket as a local file-system (e.g. S3-fuse).

If however, you are simply trying to upload a file via the shell from EC2 to S3, I would recommend Tim Kay's aws script. It is a self-contained perl script that can run without installation, or can be installed to setup alias functions. Uploading a file to an existing bucket using this script amounts to the following:

s3put bucket_name/filename.ext /path/to/local/file/name.ext

It supports most of the features of AWS and is sufficiently verbose in its error messages to make debugging a problem fairly easy.

score 2 · Answer 2 · answered Apr 23 '20 at 17:33

2

Using the aws cli commands, if you are using an EC2 Linux virtual machine and want to move files and not directories :

     aws s3 put <yourfilename> s3://<yourbucketname>/<yourdirectory>

answered Apr 23 '20 at 17:33

Nicole Douglas

121
1

score 2 · Answer 3 · answered Jul 01 '11 at 07:29

You can use s3cp tool. It's a like of scp command. For download and documentation, see http://www.beaconhill.com/opensource/s3cp.html and http://aws.amazon.com/code/Java/3124

After install, you can for example copy to S3: s3cp local-file s3://bucket/object[/]

score 2 · Answer 4 · edited May 23 '17 at 12:41

Here's a simple example to illustrate uploading directly to s3

http://sente.cc/upload_to_s3.html

code:

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  </head>
  <body>
  <h3>refresh the page after you've submitted to see your new image</h3>
    <div style="width:300px">
    <form action="http://s3.amazonaws.com/dev.sente" method="post" enctype="multipart/form-data">
      <fieldset>
      <input type="hidden" name="acl" value="public-read" /> <br />
      <i>name of key:</i><input type="text" name="key" readonly="true" value="image.jpg" /> <br />
      <input name="file" type="file" /> <br />
      <input name="submit" value="Upload" type="submit" />
    </fieldset>
    </form>
  </div>
    <br>
    <a href="http://s3.amazonaws.com/dev.sente/image.jpg">http://s3.amazonaws.com/dev.sente/image.jpg</a>
      <br>
      <a href="http://s3.amazonaws.com/dev.sente/image.jpg"><img src="http://s3.amazonaws.com/dev.sente/image.jpg"></a>
    </a>
  </body>
</html>

copied from my answer to a similar question - here

score 1 · Answer 5 · answered Jun 30 '11 at 19:56

1

Via HTTP, the same way you upload to S3 from, well, anywhere else.

answered Jun 30 '11 at 19:56

EEAA

109,363
18
175
245

If a script on the EC2 needs to get information about a image stored in the S3, the EC2 would need to request the S3 server through HTTP? Wouldn't that cost bandwidth every request and wouldn't that be slower than having the same image directly on the EC2 and requesting that image instead? Should I keep a copy of the image on the EC2 and in the S3 bucket to reduce requests made by scripts on the EC2 to the S3??? – John Jun 30 '11 at 20:06
Unless I'm mistaken, there are no transfer costs between EC2 and S3, but I may be mistaken on that. Amazon outlines things pretty well in their AWS pricing documentation. – EEAA Jun 30 '11 at 20:07
Still, wouldn't it be slower to request a image through HTTP on a different server (S3/cloudfront) rather than having the image on the same server (EC2) ? – John Jun 30 '11 at 20:11
In most cases I've see, the images are served directly out of S3 to the user, not passing through EC2 at all. Doing things that was provides a level of resiliency that EC2 can't provide. – EEAA Jun 30 '11 at 20:19
Sorry I do not understand. I'll try to clarify my question: Would it be faster or slower have a script on my EC2 web server access a image file that is already on the EC2 sever(locally) or have the EC2 server access the image file on the S3 (which I assume would be remote through HTTP, which also I assume uses bandwidth) ? – John Jun 30 '11 at 20:23
Honestly I think the difference would be negligible all things considered. Sure, S3 *may* be a bit slower, but it's a much more reliable place to store your assets than EC2. What I was getting at in my last comment is this: why do the images need to pass through the EC2 instance at all? Most sites just serve images out of S3/CloudFront directly to the users' browsers. – EEAA Jun 30 '11 at 22:39
I suppose I can save the image details in a database so I would not need to request the image itself everytime. The only thing that bothers me is that the image has to be uploaded from the EC2 (image has to be processed first - resized,etc) to the S3 sever every time because there is no way to run any server side scripts (resizing,etc) on the S3 server correct? – John Jul 01 '11 at 17:42

score 1 · Answer 6 · answered Nov 11 '11 at 13:14

In general if you're uploading files from an application to S3 for the purpose of setting up a CDN to use for that application you would want to first look and see if that application didn't already have the functionality or a plugin to do that. If you take Wordpress for instance there is the W3 Total Cache plugin that handles this for you very nicely. It is of course written in PHP and using the AWS API calls to handle the uploads/deletions when necessary.

If you're writing your own application then I would look for libraries that assist in communicating with AWS. For Python you would want to look at Boto for instance which allows you to interact with the AWS API seamlessly. As well you can always check out Amazon itself for information on the other SDK APIs that you could use.

How to upload files from Amazon EC2 server to S3 bucket?

6 Answers6