Is there a time limit for running a serverless batch job?
I followed the ETL job following this link: https://www.serverless.com/blog/etl-job-processing-with-serverless-lambda-and-redshift
I am wondering is there a limit which running the batch job?
Asked
Active
Viewed 226 times
1 Answers
0
Yes. AWS Lambda functions have time limits in their configurations and will be terminated if that is exceeded. You can change this value but only up to 15 minutes of run time (currently).
The problem is that when working with large amounts of data that is not enough time to complete some actions. In these case you will want to transition to Step Functions (state machines that can run Lambdas) and Redshift Data API (run queries w/o have a live connection all the time). In this way you can launch a long running query and have the Step Function poll for completion.
Bill Weiner
- 8,835
- 2
- 7
- 18