Amazon EMR terminateJobFlows for requests with more than maximum clusters that can be terminated

Question

I'm using AWSElasticMapReduceJavaClient-1.11.x, and the maximum clusters that EMR can terminate at one time is 10. How would I go about terminating a request with let's say 100 clusters all in one terminateJobFlows call? I'm implementing the AmazonElasticMapReduce interface so this method needs to be overridden and return a TerminateJobFlowsResult. This is unfortunately not backwards compatible with version 1.10 as terminateJobFlows was a void method, but it now returns a result.

@Override
public TerminateJobFlowsResult terminateJobFlows(TerminateJobFlowsRequest request) {
  List<List<String>> distributedClusterIds = Lists.partition(request.getJobFlowIds(), 10);

  distributedClusterIds.stream()
                .map(TerminateJobFlowsRequest::new)
                .forEach(request -> {
                    emrClient.terminateJobFlows(request); // returns a TerminateJobFlowsResult
                    // How to return each result somehow??
                });

}

On second thought, I don't really think this is possible..I'm thinking about doing the partitioning step outside of this method (where this method is called), and just call this interface method X times (in this case 10 times for 100 clusters) — Shailesh Patel, Feb 05 '19 at 22:20

score 0 · Answer 1 · answered Mar 16 '19 at 05:59

The TerminateJobFlowsResult object contains no content and is functionally equivalent to the void return in previous SDK versions. This means you can return any of the instances you receive indiscriminately or simply create a new one yourself and return it at the end to satisfy the interface contract.

Amazon EMR terminateJobFlows for requests with more than maximum clusters that can be terminated

1 Answers1