3

I have a workflow which takes a file in an S3 bucket and does a lot of processing and further requests based on the file contents. Currently, clients have to trigger the workflow manually after uploading the file. This seems to be a pretty common use case for me, so is there any way to trigger the workflow as soon as the file is uploaded?

I imagine there should be an SNS notification in between, but is there any way to send the notification directly to the SWF, without having a service consuming them and starting the workflow?

Charles
  • 50,943
  • 13
  • 104
  • 142
Pedro Werneck
  • 40,902
  • 7
  • 64
  • 85

2 Answers2

7

Update

AWS has finally launched New Event Notifications for Amazon S3 today, which indeed simply extend the long available PUT Bucket notification API with additional event types for object creation via the S3 APIs such as PUT, POST, and COPY:

  • s3:ObjectCreated:*
  • s3:ObjectCreated:Put
  • s3:ObjectCreated:Post
  • s3:ObjectCreated:Copy
  • s3:ObjectCreated:CompleteMultipartUpload

Initial Answer

[...] is there any way to send the notification directly to the SWF, without having a service consuming them and starting the workflow?

Unfortunately there is no such way, you indeed need a mediating service - while the PUT Bucket notification has obviously been designed to allow for other types of events too, Amazon S3 doesn't support Amazon SNS notifications for anything but Enabling RRS Lost Object Notifications as of today:

This implementation of the PUT operation uses the notification subresource to enable notifications of specified events for a bucket. Currently, the s3:ReducedRedundancyLostObject event is the only event supported for notifications. The s3:ReducedRedundancyLostObject event is triggered when Amazon S3 detects that it has lost all replicas of an object and can no longer service requests for that object. [emphasis mine]

Steffen Opel
  • 63,899
  • 11
  • 192
  • 211
  • 2
    Sorry, but I respectfully disagree - you asked: _[...] is there any way to send the notification directly to the SWF, without having a service consuming them and starting the workflow?_ - I provided an answer stating that there is no such way, including an explanation and a proper reference - I've added that quote to make this more obvious now. Though unfortunate, sometimes a negative answer can also be an answer (depending on the question asked). – Steffen Opel May 23 '14 at 18:38
1

As Steffen Opel said, there is no way to do this right now. However, an alternate route to what his updated answer provided would be to use AWS's new event processing service Lambda (which ATM is in preview). The documentation that shows you how to configure it for S3 is here, but at a high level:

  1. Setup a lambda function to do something on an an object-put event (in this case, call SWF)
  2. Configure bucket to publish to that lambda function with the appropriate users and roles
  3. When the event occurs to that bucket, your lambda function will be run on AWS hardware
mkobit
  • 43,979
  • 12
  • 156
  • 150