0

I am newbie and have setup Hadoop 3.3.4 on my laptop. I have performed the configurations like setting HADOOP_HOME in .bash_profile, changes to mapred-site.xml, yarn-site.xml, core-site.xml and hdfs-site.xml. I am able to start dfs and yarn and they are accessible via browser, but whenever I execute any program, I get below error.

2023-02-09 23:15:59,864 INFO mapreduce.Job: Job job_1675964716435_0001 failed with state FAILED due to: Application application_1675964716435_0001 failed 2 times due to AM Container for appattempt_1675964716435_0001_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2023-02-09 23:15:59.073]Exception from container-launch. Container id: container_1675964716435_0001_02_000001 Exit code: 1

[2023-02-09 23:15:59.077]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster

Please check whether your <HADOOP_HOME>/etc/hadoop/mapred-site.xml contains the below configuration:

  <property>
      <name>yarn.app.mapreduce.am.env</name>
      <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
  </property>
  <property>
      <name>mapreduce.map.env</name>
      <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
  </property>
  <property>
      <name>mapreduce.reduce.env</name>
      <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
  </property>

I have tried various options suggested by colleagues like setting the values in mapred-site.xml as below:

    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
        <name>mapreduce.map.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>

I have also tried to setup class path in yarn-site.xml as suggested in a different thread as:

    <property>
        <name>yarn.application.classpath</name>
        <value>[Value generated by $(hadoop class path)]</value>
    </property>

After setting individual options, I also stopped all daemon processes and started all but am unable to get through this problem.

Appreciate any help/suggestions that can be given for this!

Pranav Sheth
  • 1

0 Answers0