2

I have a problem running a map-reduce java application I simplified my problem using the tutorial code given from AWS which runs a pre-defined step:

public class Main {

public static void main(String[] args) {

    AWSCredentials credentials = getCredentials();
    AmazonElasticMapReduceClient emr = new AmazonElasticMapReduceClient(
            credentials);

    StepFactory stepFactory = new StepFactory();

    StepConfig enabledebugging = new StepConfig()
            .withName("Enable debugging")
            .withActionOnFailure("TERMINATE_JOB_FLOW")
            .withHadoopJarStep(stepFactory.newEnableDebuggingStep());

    StepConfig installHive = new StepConfig().withName("Install Hive")
            .withActionOnFailure("TERMINATE_JOB_FLOW")
            .withHadoopJarStep(stepFactory.newInstallHiveStep());

    RunJobFlowRequest request = new RunJobFlowRequest()
            .withName("Hive Interactive")
            .withAmiVersion("3.3.1")
            .withSteps(enabledebugging, installHive)
            .withLogUri("s3://tweets-hadoop/")
            .withServiceRole("service_role")
            .withJobFlowRole("jobflow_role")
            .withInstances(
                    new JobFlowInstancesConfig().withEc2KeyName("hadoop")
                            .withInstanceCount(5)
                            .withKeepJobFlowAliveWhenNoSteps(true)
                            .withMasterInstanceType("m3.xlarge")
                            .withSlaveInstanceType("m1.large"));

    RunJobFlowResult result = emr.runJobFlow(request);
    System.out.println(result);
}

   private static AWSCredentials getCredentials() {
        AWSCredentials credentials = null;
        credentials = new BasicAWSCredentials("<KEY>","<VALUE>");
        return credentials;
    }

}

where , are the secret active key and 'hadoop' is a keypair I created in the EC2 console.

After running I see the Job trying to start in the EMR console, after 1 minute it changes from 'starting' to 'Terminated with errors Validation error'

no other information is given

Any ideas what goes wrong?

Thanks!

user3537890
  • 21
  • 1
  • 2
  • For more clues look at the EMR log folder on S3 that you have configured. As a first step, precisely follow the steps outlined in the tutorial http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/gsg-launch-cluster.html . – user1452132 Feb 28 '15 at 17:06

2 Answers2

2

You can check the details of error on your EMR cluster list detail page (on the top). Because it is a validation error it has not come in logs yet, so the only way is check the exception details/cli response/aws console...

My guess is that instance types you used are not supported in EMR (they are supported in EC2 but not in EMR). However, you will get exact problem once you follow the given step.

Naveen Sachar
  • 454
  • 4
  • 8
  • 1
    Those instance types (m3.xlarge, m1.large) are supported in AWS EMR clusters. - m1.medium 1 3.8 410 SSD - m1.large 2 7.5 850 SSD - m1.xlarge 4 15 1690 SSD - m2.xlarge 2 17.1 420 SSD - m2.2xlarge 4 34.2 850 SSD - m2.4xlarge 8 68.4 1690 SSD - m3.xlarge 8 15 80 SSD - m3.2xlarge 16 30 160 SSD - m4.large 4 8 EBS only etc – williambq Mar 21 '17 at 20:29
0

I saw an error in steps saying no logging exists hence step cancelled which showed up in console as terminated due to validation error. So I enabled logging and it seems working now

Vipin
  • 1