2

I need to write a map reduce that takes input as two input files. First input file looks like this:

key1 , 25
key1 , 35
key1 , 60
key2 , 30
key3 , 45
key3 , 65

Second input file is as follows:

key1, -10
key2, -20
key3, -15

and I need to get an output as:

key1 , 15
key1 , 25
key1 , 50
key2 , 10
key3 , 30
key3 , 50

(The output is first input file's values subtracted by the second input file)

How could this be done? How will the mapper and reducer task look like?

My approach is as follows:

I think I will have to have two mappers, one per input file (Can a single mapper be used to read both the files?). Mappers will simply emit the key and the value.

At the reducer end, when I receive all values corresponding to a key, I have to subtract the values, that is coming from the first file, by the value in the second file.

So I need to find out whether the corresponding value is coming from the second input file or first file. how can this be done?

Any other better approaches?

Hamid Rouhani
  • 2,309
  • 2
  • 31
  • 45
user2715182
  • 653
  • 2
  • 10
  • 23
  • One map, if key not exists just create map entry, otherwise add or subtract key value from from existing map entry – krzydyn Aug 14 '15 at 14:38

2 Answers2

1

Read in a separate mapper, and alter the contents so that you know which file they come from. e.g. output

key1 , 25 , file1
key1 , 35 , file1
key1 , 60 , file1
key2 , 30 , file1
key3 , 45 , file1
key3 , 65 , file1

key1, -10 , file2
key2, -20 , file2
key3, -15 , file2

Then, you can both outputs through a single mapreduce phase together, and you will know which is from where, and you can manipulate your data accordingly in your reducer.

mattinbits
  • 10,370
  • 1
  • 26
  • 35
1

This can be done in a single MapReduce program. You can use MultipleInputs support from MapReduce framework.

  • Define two mapper classes for each input file. Then output key, value as key#fileName, value pair.
  • Define a custom partitioner, which consider only actual key and ignore the appended fileName for partitioning the data. So that the same keys from both files goes to the same reducer.

  • The reducer will get the list of values for key from file1. Hold this list of values in memory and fetch the list of values from file2 as well for the same key. These two will come consecutively because we have partitioned the data on only key part and comparator will also sort them on key value. Assuming the first file name comes alphabetically. Then perform them required operation on first file value list using second file value.

    Configuration conf = new Configuration();
Job job = new Job(conf, "aggprog");        

MultipleInputs.addInputPath(job,new Path(args[0]),TextInputFormat.class,MapperOne.class);
MultipleInputs.addInputPath(job,new Path(args[1]),TextInputFormat.class,MapperTwo.class);

conf.setPartitionerClass(CustomPartitioner.class);

    Hope this helps.

YoungHobbit
  • 13,254
  • 9
  • 50
  • 73