Best way to 'listen' to s3 bucket for new file?

Question

I have connected zapier to a webhook I am listening too, which sends a JSON file into my s3 bucket.

I have some python code that I want to execute when a new file is uploaded into the bucket, in real time over the file.

What is the best way to 'listen' for the upload of this file into the s3 bucket?

Have you considered using a lambda function? https://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html#supported-event-source-s3 — Liam Foley, Nov 25 '18 at 19:43
@LiamFoley I have not thought about using a lambda function, open to all suggestions. Something that is easy to setup is exactly what I am looking for. — RustyShackleford, Nov 25 '18 at 19:44

score 5 · Accepted Answer · answered Nov 25 '18 at 19:46

5

You can setup an Amazon's CloudWatch Event to listen for when a new object is put into a S3 bucket. You can configure the event to work with a specific S3 bucket. When a new object is put into the bucket you can configure the event to trigger a Lambda function that will execute whatever custom Python code you want to run when a new object is added.

Your question is pretty broad so I can't get into specifics, but you can checkout the AWS documentation that provides examples to guide you through the process. AWS CloudWatch Documentation.

answered Nov 25 '18 at 19:46

vielkind

2,840
1
16
16

do you think I could read in the file from the s3 bucket into aws lambda for the python bit? – RustyShackleford Nov 25 '18 at 19:47
Yes. If you configure a Lambda function to get triggered by a Put event details about the object such as the bucket and key can be passed to the Lambda function. You can leverage that information to read the file that has been uploaded. – vielkind Nov 25 '18 at 20:10

score 2 · Answer 2 · answered Nov 26 '18 at 20:06

2

David here, from the Zapier Platform team.

Seems like you've already found your answer, which is great. I just wanted to plug Zapier as an option (since you had mentioned you're already using it). Our S3 integration has a "new file in bucket" trigger, which you can combine with any other step (such as a Python Code step). Additionally, you could skip the middleman and structure your zap as:

some trigger
Add file to S3
Run Python

And not need to worry about webhooks at all.

Let me know if you've got any other questions!

answered Nov 26 '18 at 20:06

xavdid

5,092
3
20
32

thank you for the reply! I would love to keep everything under one roof. Currently I have setup zapier webhook to s3 then my code. My code has libraries such as pandas, requests, and json to name a few. Would these dependencies run on zapier platform? – RustyShackleford Nov 29 '18 at 00:29
Not as Python, no. Unfortunately we don't support python libraries in code steps. You can use JS libraries in a CLI app, but that's a pretty substantial re-write that you don't need to do. In this case, it sounds like you could still benefit from the S3 trigger, but the action would send a webhook to wherever your code is. – xavdid Nov 30 '18 at 15:23

score 0 · Answer 3 · answered Nov 25 '18 at 19:41

0

There are a few recommended options involving event notifications in S3 buckets outlined here in AWS docs https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-event-notifications.html

answered Nov 25 '18 at 19:41

TheArchDev

53
5

Thank you, in your opinion what is the easiest way? – RustyShackleford Nov 25 '18 at 19:50

Best way to 'listen' to s3 bucket for new file?

3 Answers3