-1

I have a .py file with code in a AWS Cloud9 environment.

I want to run this code when a file is uploaded to a specific S3 Bucket.

But, it seems that I only can have triggers with AWS Lambda functions. I don't know how to add triggers to the AWS Cloud9 functions...

My simple code in the .py file in the Cloud9 environment is:

import boto3
import os
import sys

s3 = boto3.client('s3')

s3.download_file('our-awesome-first-test-bucket', 'test_text.txt', 'test_text_saved_to_env.txt')

with open('test_text_saved_to_env.txt', 'r') as f:
    output = sum(map(int, f))
   
with open('output_to_awesome_bucket.txt', 'w') as outf:
    outf.write(str(output))
   
s3.upload_file('output_to_awesome_bucket.txt', 'outputbucket-for-our-first-awesome-bucket', 'output_to_awesome_bucket.txt')

os.remove('test_text_saved_to_env.txt')
os.remove('output_to_awesome_bucket.txt')

How can I trigger this code to run when a file is uploaded to the S3 Bucket?

U13-Forward
  • 69,221
  • 14
  • 89
  • 114
  • Cloud9 is an IDE. You deploy from there to your target environment e.g. Lambda if you want an event-driven solution. – jarmod Mar 26 '22 at 16:55
  • @jarmod I kept trying to do that as you mentioned. But, I couldn't create an AWS Lambda function from inside the Cloud9 IDE console, because, to create a Lambda function that way requires Python 3.6. I am fine with that, but, my Cloud9 Python default version is set to 3.7.10, and I can't change it to 3.6... Therefore, it throws an error. – U13-Forward Mar 26 '22 at 23:46
  • Did my solution work for you? – Paolo Apr 02 '22 at 19:34

1 Answers1

0

As mentioned in the documentation, S3 supports the following destinations as a trigger once an object has been uploaded:

  • Amazon Simple Notification Service (Amazon SNS) topics
  • Amazon Simple Queue Service (Amazon SQS) queues
  • AWS Lambda function

The ideal solution to use would be to simply run a lambda function, but as you mentioned you have dependencies on libraries which have a size greater than 250MB, which is the limit for lambda.

In this case, you should consider packaging up your Python application in a Docker image which you can run in an ECS task. You should also create a lambda function as a wrapper which will invoke the ECS task.

Configure the bucket to trigger the lambda which, in turn, will run the ECS task.

As an alternative, if you wish to run the script on the Cloud9 EC2 instance, you can configure a service on the EC2 instance which polls an SQS queue (configured as a target for your S3 bucket) and then runs your script if an event has been detected in the queue.

Paolo
  • 21,270
  • 6
  • 38
  • 69
  • Thanks for your answer. But, the reason I am not using AWS Lambda already, is because the size of all the libraries that I need for executing my function are exceeding the maximum size allowed to upload to a AWS Lambda Layer. – U13-Forward Mar 26 '22 at 23:39
  • @U12-Forward What is the unzipped size of the libraries you need? – Paolo Mar 26 '22 at 23:57
  • What I need is over the limit of 250MB. – U13-Forward Mar 27 '22 at 00:00
  • I have no idea how to use Docker images... And now I don't have the time to run it, it's on my list for the future... But for now, I would like to just use and run this code with Cloud9... Is there any other way? – U13-Forward Mar 27 '22 at 00:16
  • @U12-Forward I've provided an alternative for you. You can configure the S3 bucket to send notifications to an SQS queue, and you can create a service on the EC2 instance (where your Cloud9 environment is running) so that it polls the queue for new events. If an event has been detected in the queue, then you run your Python script. – Paolo Mar 27 '22 at 10:38