3

I am writing a mapreduce program which has 2 mappers and 1 reducer, I implemented custom Writable Datatypes for each mapper. The Datatype is more or less just a container where the fields are Text / Intwritable values.

So Mapper 1 outputs id(Text), M1Writable (my writable with 3 fields)

Mapper 2 outputs id(Text), M2Writable (my writable with 2 fields)

The reducer will get an iterable values

how will this work since the two types of writable are different?

How can I determine which mapper it came from?

Thanks I know this is a basic question, but I was having issues finding an answer.

VSEWHGHP
  • 195
  • 2
  • 3
  • 12
  • Why bother thinking about multi mapper? Why not just use one mapper, and move your output logic to the mapper. In short the output logic is in one mapper, the mapper can output the expected outputs as you need. – luoluo Oct 29 '15 at 03:10
  • Are you trying to achieve reduce side join with two data sets? – Ramzy Oct 29 '15 at 13:14
  • You guys are right, It's just better logic to either save my mapper output or have an abstract writable class which my Output writables extend so they are the same parent type. Thanks guys – VSEWHGHP Oct 30 '15 at 03:19

1 Answers1

2

Mappers in MapReduce always output the same type.

Unless M1Writable and M2Writable have a common parent class (e.g., MWritable), which is the common output type of all mappers, you cannot output different types among your mappers.

To know which mapper the output came from, you need to record the information in your custom MWritable objects.

PNS
  • 19,295
  • 32
  • 96
  • 143