0

I am trying to run an insert query and face following error using mapreduce

Application application_1609169302439_0001 failed 2 times due to AM Container for appattempt_1609169302439_0001_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2020-12-28 16:29:05.332]Exception from container-launch. Container id: container_1609169302439_0001_02_000001 Exit code: 1 [2020-12-28 16:29:05.335]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster Please check whether your <HADOOP_HOME>/etc/hadoop/mapred-site.xml contains the below configuration: yarn.app.mapreduce.am.env HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory} mapreduce.map.env HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory} mapreduce.reduce.env HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}

while looking at my mapred-site.xml config file

<configuration>
 <property>
  <name>mapreduce.jobtracker.address</name>
  <value>localhost:54311</value>
 </property>
 <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
 </property>
 <property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=$HADOOP_MAPRED_HOME</value>
 </property>
 <property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=$HADOOP_MAPRED_HOME</value>
 </property>
 <property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=$HADOOP_MAPRED_HOME</value>
</property>
</configuration>

My understanding is that it is a configuration issue but cannot find any clear and simple answer what is missing on my sytem.

I had installed tez before but it wasn't working either.

Any help or guidance would be apreciated. I browsed site and could find similar issues reported but wasn't able to fix mine based on solution provided.

Best

user2448864
  • 1
  • 1

1 Answers1

0

After some research, I updated my configuration and I am now able to run mapreduce job from hive cli

Here's updated part of mapred-site.xml

> <property>   <name>yarn.app.mapreduce.am.env</name>  
> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> <property> 
> <name>mapreduce.map.env</name>  
> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> <property> 
> <name>mapreduce.reduce.env</name>  
> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> <property> 
>     <name>mapreduce.application.classpath</name>
>     <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/common/*,$HADOOP_MAPRED_HOME/share/hadoop/common/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/lib/*</value>
> </property>

yarn-site.xml also contains the path to jars.

 <property>
       <name>yarn.application.classpath</name>
       <value>
            %HADOOP_HOME%\etc\hadoop,
            %HADOOP_HOME%\share\hadoop\common\*,
            %HADOOP_HOME%\share\hadoop\common\lib\*,
            %HADOOP_HOME%\share\hadoop\hdfs\*,
            %HADOOP_HOME%\share\hadoop\hdfs\lib\*,
            %HADOOP_HOME%\share\hadoop\mapreduce\*,
            %HADOOP_HOME%\share\hadoop\mapreduce\lib\*,
            %HADOOP_HOME%\share\hadoop\yarn\*,
            %HADOOP_HOME%\share\hadoop\yarn\lib\*
       </value>
    </property>

After these updated and restarting the cluster, I am able to run jobs and follow their progress on port 8088

Best

user2448864
  • 1
  • 1