1

I have the following set in my mapred-site.xml in my ec2 inctance where my Hadoop is working :

<property>
  <name>mapreduce.map.memory.mb</name>
  <value>2048</value>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>4096</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx1024m</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx3072m</value>
</property>

With the above properties set Sqoop works perfectly but Hive Tez doesnot. So I changed it to the following:

<property>
 <name>mapreduce.map.memory.mb</name>
 <value>4096</value>
</property>
<property>
 <name>mapreduce.reduce.memory.mb</name>
 <value>8192</value>
</property>
<property>
 <name>mapreduce.map.java.opts</name>
 <value>-Xmx3072m</value>
</property>
<property>
 <name>mapreduce.reduce.java.opts</name>
 <value>-Xmx6144m</value>
</property>

as Hive Tez needs more memory, Sqoop stops working and hive Tez gives me error like:

Diagnostic Messages for this Task:
Container [pid=51661,containerID=container_e50_1493005386967_25486_01_000243] is running beyond virtual memory limits. Current usage: 274 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container

Suggest me how can I make both Sqoop and Hive tez work.

0 Answers0