Save imagemagick file directly to s3 from ec2

Question

I download an image from a url to an ec2 instance(wget), process it with imagemagick (convert -actions $savedfilename) and then save it to s3 (with a php api).

Is it possible to process the image on ec2 and save it directly to s3 without writing to the s3 volume?
If this is possible, does it make sense - would it be faster / cheaper?
Is there anything else that can be done to improve the efficiency of the process?
Is there an API to save to s3 directly from the shell?

score 2 · Accepted Answer · answered Jun 16 '12 at 11:23

I guess you meant "...without writing to the EBS volume" Am I right? You can pipe Wgets output directly to ImageMagicks convert, which looks like this:

wget -O - 'http://d24w6bsrhbeh9d.cloudfront.net/photo/4498158_700b_v1.jpg' | convert - test.png

Take a look at s3cmd, it will allow you to interact with S3 directly from the command line. Our example workflow will then look like this:

wget -O - 'http://d24w6bsrhbeh9d.cloudfront.net/photo/4498158_700b_v1.jpg' | convert - test.png && s3cmd put --acl-public --guess-mime-type test.png s3://example.com/images/test.png

This will give you this result, which you can filter using regex to get the public URL:

File 'test.png' stored as s3://example.com/images/test.png (xxxx bytes)
Public URL of the object is: http://example.com.s3.amazonaws.com/images/test.png

Get URL from text:

<?php
  $reg_exUrl = "/(http|https|ftp|ftps)\:\/\/[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,3}(\/\S*)?/";
  $cmd_output = "File 'test.png' stored as s3://example.com/images/test.png (xxxx bytes) Public URL of the object is: http://example.com.s3.amazonaws.com/images/test.png";
  if(preg_match($reg_exUrl, $cmd_output, $url)) {
    $image_url = $url[0];
  }
  else {
    // no url found …
  }
?>

I guess that's an elegant way of doing your process :) I'm not sure if it will be any faster or cheaper … Maybe a bit because of EBS' bad disk I/O.

One of my imagemagick commands has two steps to it and requires that I save the result of the first step before applying the second. Is it possible to save the result of step one in memory instead of writing to the volume and access it from memory in step two? — waigani, Jun 18 '12 at 00:34
Post some code, then I can help you. Or take a look at this discussion: http://www.imagemagick.org/discourse-server/viewtopic.php?f=1&t=20906 — dom, Jun 18 '12 at 06:37
Okay, its all working except I need to pipe the output from s3cmd. I've made a new question: http://stackoverflow.com/questions/11118671/how-do-you-pipe-the-result-of-s3cmd-get-to-a-var — waigani, Jun 20 '12 at 11:37

score 0 · Answer 2 · answered Aug 11 '17 at 15:36

From the aws s3 cli cp documentation:

Uploading a local file stream to S3

WARNING:: PowerShell may alter the encoding of or add a CRLF to piped input.

The following cp command uploads a local file stream from standard input to a specified bucket and key:
aws s3 cp - s3://mybucket/stream.txt

So similar to @dom's answer you could do

wget -O - 'http://d24w6bsrhbeh9d.cloudfront.net/photo/4498158_700b_v1.jpg' | convert - test.png | aws s3 cp - s3://example.com/images/test.png --acl public

Save imagemagick file directly to s3 from ec2

2 Answers2