0

i need help regarding map reduce chaining.i have a map reduce chain like this

map->reduce->map

i want the output of reducer to be used in the last mapper

for example, in my reducer i am getting the max salary of an employee and this value is supposed to be used in the next mapper and find the record with that max salary value.so obviously my last mapper should get the output of the reducer and the contents of the file?is it possible?how can i fix the problem?any better solution?

user1585111
  • 1,019
  • 6
  • 19
  • 35

1 Answers1

0

I'm not sure i understood the problem, but i will try to help.
You have reduced some input containing an employee salaries (lets call it input1) into output (lets call it output1) that looks like that:
Key: someEmployee Value: max salary.
and now you want another mapper to to map the data from both input1 and output1?
if so, than u have few options, u may choose one according to your needs.

  • Manipulate first reducer output. instad of creating output1 in the format
    Key: someEmployee Value: max_salary@@salary_1,salary_2,salary_3...salary_n
    and than create new job, and set the new mapper input as output1.

  • Try reading this issue explaining how to get multiple inputs into one mapper

Community
  • 1
  • 1
Roy
  • 221
  • 3
  • 8
  • Sorry iam new to hadoop,so struggling with it.i will find the max salary in my reducer,and this max salary,contents of the file should be the input to the next mapper,so that i will find records with max salary. – user1585111 Aug 19 '13 at 11:31
  • So the first solution should be exactly what you need. just create another mapreduce job, and enter the output file u generated at the first step as an input. – Roy Aug 19 '13 at 12:19
  • Did my answer helped you? do you need anything else? if not, please mark is as an answer. – Roy Aug 20 '13 at 09:02
  • how to set output1 to new mapper,because mapper input will be set like this right FileInputFormat.setInputPaths(jobConf, new Path("hdfs://localhost:9000/employee_da – user1585111 Aug 20 '13 at 11:29