Hadoop Streaming Python Multiple Input Files Single Mapper

Question

I have a single mapper.

for line in sys.stdin:
    #if line is from file1
    #process it based on some_arbitrary_logic
    #emit k,v

    #if line is from file2
    #process it based on another_arbitrary_logic
    #emit k, v

And I need to call this mapper through a hadoop streaming API with -input file1 and another -input file2.

How do I achieve this? How do I know which line belongs to which file in the STDIN that hadoop streaming gives me?

UPDATE

File1

Fruit, Vendor, Cost

Oranges, FreshOrangesCompany, 50
Apples, FreshAppleCompany, 100

File2

Vendor, Location, NumberOfOffices

FreshAppleCompany, NewZealand, 45
FreshOrangeCompany, FijiIslands, 100

What I need to do is print out in how many offices do they sell oranges.

Oranges 100.

So both files need to be INPUT to the mapper.

The data coming through stdin is a new data source than file1 and file2? Or do you want to run your mapreduce job over file1 and file2? — Donald Miner, Dec 18 '13 at 14:28
@DonaldMiner See this http://stackoverflow.com/a/4839045/2358206 — ComputerFellow, Dec 18 '13 at 14:35
@DonaldMiner Hadoop streaming gives the mapper as stdin what it receives as files, that's what http://stackoverflow.com/users/37751/jd-long explains there. — ComputerFellow, Dec 18 '13 at 14:37
You pass in the input files with the `-input` flag, not the `-file` flag. The `-file` flag is for reference data files or the mapper code itself. There is a big difference between the files in `-input` getting passed via stdin and `-file` just shipping the file to the local working directory. — Donald Miner, Dec 18 '13 at 16:00

score 5 · Answer 1 · answered Dec 18 '13 at 15:21

5

os.environ["map.input.file"] inside the mapper python code should give the file name of the block which the mapper is processing.

answered Dec 18 '13 at 15:21

Praveen Sripati

32,799
16
80
117

1

Do you know where this is documented? I did a quick search and couldn't find anything official. I just learned of this today...! Thanks! – Donald Miner Dec 18 '13 at 16:01
Check the updateJobWithSplit in [here](http://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.4/src/mapred/org/apache/hadoop/mapred/MapTask.java). It's not there in the documentation, but in the Hadoop - The Definitive Guide. – Praveen Sripati Dec 18 '13 at 16:31

score 1 · Answer 2 · edited May 23 '17 at 12:21

The question is slightly ambiguous, in the sense that not much details was provided as far as the purpose of the files are concerned. So I'm making some assumptions

If the file1 and file2 are just two sets of data file and have same kind of data and all you need to ensure is that the files are processed... then just copy the files to a HDFS folder and ensure that the folder is identified as input folder and you are good. Data from both files will be used to call mappers...
If it is that file1 and file2 have different purpose. For example, file1 is input file for mapper but file2 is something that you need to refer to for some joins or etc...then distributed cache. Check this Hadoop Streaming with multiple input
If file1 and file2 are both input files and are related and you need to do a join. If file1 or file2 is small, then you might be able to use it as distributed cache using regular file or archive file. However, if both the files are big, then it is slightly complicated as you may have to do multiple MR or do an transformation of the file to a format that can be used by Hive and use a hive join and then use the join result as input to your Python Mapper job.

Hope this helps.

Hadoop Streaming Python Multiple Input Files Single Mapper

2 Answers2