I am new to Amazon simple workflow service. Is there a way to run the swf workflows on EMR. I have AWS CLI setup and able to bootstrap hadoop and bring up the cluster. I have not found enough documentation on this and no source on the web. Is there any change that I can boot the EMR cluster using SWF instead of AWS CLI. Thanks.
Asked
Active
Viewed 301 times
1 Answers
0
You should use one of the dedicated AWS SDKs to coordinate between the two services. I am successfully using the AWS SDK for Java to create a workflow that starts several EMR clusters in parallel with different jobs and then just waits for them to finish, failing the whole workflow if one of the jobs fail.
Out of all available AWS SDKs, I highly recommend the Java one. It struck me as extremely robust. I have also used the PHP one in the past, but it lacks on certain departments (it does not provide a 'flow' framework for SWF for example).
Ana Todor
- 781
- 1
- 6
- 15