2

What I am trying to accomplish is building the simplest system to process image uploading through message queues.

Right now we are working with temporary files, we build one for each image that needs to be sent over Aws S3 and optimize/customize it depending on our needs. Once all of this is done, we push it to S3.

Now, this currently works, but there's the filesystem overhead that I would like to remove by using base64 encodings of the image, making the process entirely decoupled from the system where the application is running.

Since we are going to use Amazon SQS (we currently use Beanstalkd on production), their service does not allow for more than 256Kb of payload to be pushed over the queue, and this is a problem since images are pretty heavy.

What solutions are still available to explore?

GiamPy
  • 3,543
  • 3
  • 30
  • 51
  • Send the intermediate file to S3 and reference its temporary location url in the SQS message, maybe? – Michael - sqlbot Nov 09 '16 at 13:28
  • That's an idea, if only the PHP SDK supported it... I've seen AWS developed an extended client but only for Java. Grrr. – GiamPy Nov 09 '16 at 14:36
  • The SDK doesn't automagically support it with SQS, but there isn't any obvious reason why you couldn't just store the file in S3 yourself and include the URI in the SQS message. – Michael - sqlbot Nov 09 '16 at 20:07
  • I surely can! There's an overhead of using two libraries though. An extended client for PHP (just like the one that exists on Java) that automatically recognizes the payload size would help to speed up the feature development. – GiamPy Nov 09 '16 at 20:17
  • Also, since the context is image uploading, I'd need to store the temporary file (which the user has uploaded) in S3, instead of local filesystem, so that is ready to be used in SQS. This forces me to PUT the object on S3 synchronously, adding some latency for every picture that's being uploaded. – GiamPy Nov 09 '16 at 20:23
  • 1
    True, though within EC2 S3 has a very fast connection. Alternately, the user could [directly upload it to S3](http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-UsingHTTPPOST.html) and the success URL redirect could take them to your page where you enqueue to SQS. – Michael - sqlbot Nov 09 '16 at 20:28
  • @Michael-sqlbot That is new for me, direct upload. A good one definitely. I'll look into it. Post it as an answer! – GiamPy Nov 09 '16 at 20:31

1 Answers1

1

You can look at breaking the file into suitable message sizes, 256Kb in this case, tag the messages with appropriate sequence numbers and put to queue. At the other end, receive the messages re-assemble the messages in the order and write to a file or push image data to further stages in your solution.

Shashi
  • 14,980
  • 2
  • 33
  • 52
  • 2
    SQS only accepts valid UTF-8 as payload, not binary data, so the content would have to be base64-encoded for transport. Potential out of order delivery also makes this a little bit fragile. – Michael - sqlbot Nov 09 '16 at 13:28
  • Okay. Thanks. But isn't there a way/config option to ensure delivery order? – Shashi Nov 09 '16 at 13:34
  • 1
    No. AWS SQS does not ensure that the order with which you've sent the jobs is the same when you're going to receive them. You need to handle it at application level. – GiamPy Nov 09 '16 at 14:35
  • Refer this link. https://aws.amazon.com/about-aws/whats-new/2015/10/now-send-payloads-up-to-2gb-with-amazon-sqs/#:~:text=Amazon%20Simple%20Queue%20Service%20(SQS,payloads%20were%20limited%20to%20256KB. – Krishantha Dinesh Mar 24 '22 at 11:31