1

Say I have 100 mappers running in parallel and there are total 500 mappers running.

Input size received by each mapper is almost same and the processing time each mapper should take should be more or less identical.

But say first 100 mappers finishes in 20 minutes, the next 100 mappers take like 25-30 minutes and the next batch of 100 mappers take around 40-50 minutes each. And then later we get GC overhead error.

Why is this happening?

I have following configurations already set:

  <property><name>mapred.child.java.opts</name><value>-Xmx4096m</value></property>
  <property><name>mapred.job.reuse.jvm.num.tasks</name><value>1</value></property>

What else can be done here?

Amar
  • 11,930
  • 5
  • 50
  • 73
  • If you get GC errors, you are likely creating too much objects, or your heap is not enough. Maybe you have a memory leak. – Thomas Jungblut Nov 07 '12 at 08:05
  • Yeah but if that's the case it should be failing at the first run only, right? Why it slows down and fails on later runs? Also since JVM reuse is set to 1, each mapper would get a fresh one to use! – Amar Nov 07 '12 at 16:30
  • Why runs? Each of your mappers have different input, so it's not unlikely that a malformed input causes memory leaks. If you set reuse to 1, you are reuising a JVM a single time. Set it to -1 then you will reuse every process. However this won't solve your memory leak. – Thomas Jungblut Nov 07 '12 at 16:35
  • Bu runs I meant batch of mappers, like I had mentioned above that first 100 mappers run properly then for next batch of mappers it starts getting slower and finally fails throwing GC overhead limit exceeded limit. So there is something else happening here other than malformed input and memory leak. – Amar Nov 07 '12 at 16:41
  • Can you share your mapper code, or describe what your doing in your map stage - the input sizes might be the same, but the logic of your mapper and the actual input contents could be a cause (like if you were buffering output values by some key for example) – Chris White Nov 10 '12 at 13:37

0 Answers0