2

I'd like to peform the following tasks on a regular basis (e.g. every day at 6AM) using AWS:

  • get new set of data using API. This dataset is updated on a daily basis.
  • run a python script that would process the obtained dataset by the means of several python libraries like matplotlib, pandas, plotly
  • automatically send the output of the script, which would be a single pdf file or a html dashboard, via email to a group of specified recipients

I know how to perform all of the above items locally - my goal is to automate this routine. I'm new to AWS and would appreciate some advice on how to perform these tasks in a straightforward way. Based on the reading I did so far, it looks like the serverless approach may be able to do the job and also reduce the complexity, but I'm not sure which functionalities exactly I should use.

Thomas
  • 31
  • 2
  • 1. check out SAM. it makes deploying lambdas easier. 2. your gonna need to use a layer for your libraries, and you'll need to package them on linux, as ones you package on windows won't work on lambda. 3. the boto3 library will help you – Neil McGuigan Apr 06 '21 at 22:15

2 Answers2

1

For scheduling you can use aws event bridge. You can schedule AWS lambda or AWS Step Functions both of these are serverless :).

You can have 3 lambdas

  1. To get the data and save it in S3/dynamo (if you want to persist the data)
  2. Processor lambda and save the report to S3.
  3. Another lambda to send email using AWS SES which will read the report from S3 and send it.

If you don't want to use step function you can start your lambda from S3 put event or you can trigger one lambda from another lambda using aws-sdk.

nirvana124
  • 903
  • 1
  • 9
  • 12
0

So there are different approaches you can take.

First off, I would create a Lambda. You can schedule the function to run on a cron job.

If the Message you want to send is small:

I would create a SNS Topic with a email fan out.

Inside your lambda you can then transform the data and send out via SNS.

Otherwise:

I would use SES and send a mail via the SES SDK.

Robert Kossendey
  • 6,733
  • 2
  • 12
  • 42