2

I'm running a hadoop job on Amazon Elastic MapReduce and I keep getting an OutOfMemory error. The values are admittedly a little bit larger than most MapReduce values, but it seems even when I decrease the size dramatically it still happens. Here's the stack trace:

Error: java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1698)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1558)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1407)
    at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1339)

I'm not exactly sure what code to show since this stack trace is entirely outside the scope of my code. The version is hadoop 0.20.205

Is there some way to configure the reducer to read less values at a time? Shouldn't that be handled automatically based on the available memory?

dspyz
  • 5,280
  • 2
  • 25
  • 63
  • How many reducers are configured? – Thomas Jungblut Jan 23 '12 at 18:58
  • 105, but why should that matter – dspyz Jan 23 '12 at 19:09
  • 3
    Because OOM errors may normally occur when your reducer count is too small. So the inmemory shuffle goes out of heap. You can either increase the heap of the reduce tasks or increase the number of reducers. – Thomas Jungblut Jan 23 '12 at 21:13
  • EDIT: Never mind, I think I understand – dspyz Jan 23 '12 at 21:38
  • Please don't delete your comment ;) If you have a question, feel free to ask. That's what we are here for. – Thomas Jungblut Jan 23 '12 at 21:52
  • Alright, as I understand it. Hadoop loads into memory all the bytes which will be read in by each Reducer before instantiating the Reducer. So if I'm processing a very large amount of data, then I must use enough reducers so that each one won't be processing more bytes than can fit in memory. This means when dealing with terabytes, I should expect the number of reducers to be in the thousands (or perhaps even tens of thousands) – dspyz Jan 23 '12 at 22:06
  • Were you guys ever able to solve this? I'm having the same problem and would appreciate any advice. :) – Suman Feb 12 '13 at 22:29
  • 1
    Yes, I just increased the number of reducers. Make sure there are enough reducers that when you divide the size of the data by the number of reducers, you get something that fits comfortably in memory. – dspyz Feb 13 '13 at 00:11

0 Answers0