I want to use EMR spot instances to cut down my Redshift and aws glue costs, but after reading about them I want to know if I am running a 30 mins jobs how likely is it to get interrupted , How often these spot instances are taken away while running a Job and if they are taken away how can I manage my job to re-run again.
Mostly my focus is on spark job.
Opinion-based, but here goes.
Excellent read: https://aws.amazon.com/blogs/big-data/spark-enhancements-for-elasticity-and-resiliency-on-amazon-emr/
Basically AWS allow you to use spot instances and recover gracefully due to integration with YARN’s decommissioning mechanism. You need code nothing in your Spark App.
That said, if you are wanting to run using Spot Instances, you can wait for the output, but it may take a while.
AWS Glue is serverless and hence has nothing to do with EMR. Redshift is also costed differently.
