elastic map reduce "keep alive" specification in the java api

Question

How do I set the jobflow to "keep alive" in the java api like I do with command like like this:

elastic-mapreduce --create --alive ...

I have tried to add withKeepJobFlowAlivewhenNoSteps(true) but this still makes the jobflow shut down when a step fails (for example if I submit a bad jar)

score 1 · Answer 1 · answered Aug 20 '13 at 18:16

You need to set withActionOnFailure to let the API know what to do when a step fails, and this has to be set on per step basis.

You must be having withActionOnFailure("TERMINATE_JOB_FLOW") for your StepConfigs. Change them to withActionOnFailure("CANCEL_AND_WAIT").

Following is the full code to launch a cluster using Java API taken from here, just replacing the needful:

AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
   AmazonElasticMapReduceClient emr = new AmazonElasticMapReduceClient(credentials);

   StepFactory stepFactory = new StepFactory();

   StepConfig enabledebugging = new StepConfig()
       .withName("Enable debugging")
       .withActionOnFailure("CANCEL_AND_WAIT") //here is the change
       .withHadoopJarStep(stepFactory.newEnabledebuggingStep());

   StepConfig installHive = new StepConfig()
       .withName("Install Hive")
       .withActionOnFailure("CANCEL_AND_WAIT") //here is the change
       .withHadoopJarStep(stepFactory.newInstallHiveStep());

   RunJobFlowRequest request = new RunJobFlowRequest()
       .withName("Hive Interactive")
       .withSteps(enabledebugging, installHive)
       .withLogUri("s3://myawsbucket/")
       .withInstances(new JobFlowInstancesConfig()
           .withEc2KeyName("keypair")
           .withHadoopVersion("0.20")
           .withInstanceCount(5)
           .withKeepJobFlowAliveWhenNoSteps(true)
           .withMasterInstanceType("m1.small")
           .withSlaveInstanceType("m1.small"));

   RunJobFlowResult result = emr.runJobFlow(request);

elastic map reduce "keep alive" specification in the java api

1 Answers1