Hadoop Map Task : Read the content of a specified input file

Question

I'm pretty new to Hadoop environment. Recently, I run a basic mapreduce program. It was easy to run.

Now, I've a input file with following contents inside input path directory

fileName1
fileName2
fileName3
...

I need to read the lines of this file one by one and create a new File with those names (i.e fileName1, fileName2, and so on) at specified output directory.

I wrote the below map implementation, but it didn't work out

public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter)
                throws IOException {

            String fileName = value.toString();
            String path = outputFilePath + File.separator + fileName;
            File newFile = new File(path);

            newFile.mkdirs();
            newFile.createNewFile();
        }

Can somebody explain me what I've missed out ?

Thanks

Please do read the basics of MR/Hadoop and do some practice and post a query. — Praveen Sripati, Oct 15 '13 at 11:14
@PraveenSripati : Please let me know what needs to be done here.. — hadoopDev, Oct 15 '13 at 11:44

score 0 · Answer 1 · answered Oct 15 '13 at 11:35

I think you should get started with studying the FileSystem class, I think you can only create files in the distributed filesystem. Here's a code example of where I opened a file for reading, you probably just need a FSDataOutputStream. In your mapper you can get your configuration out of the Context class.

    Configuration conf = job.getConfiguration();
    Path inFile = new Path(file);
    try {
        FileSystem fs;
        fs = FileSystem.get(conf);

        if (!fs.exists(inFile))
            System.out.println("Unable to open settings file: "+file);

        FSDataInputStream in = fs.open(inFile);
                    ...
    }

score 0 · Answer 2 · answered Oct 15 '13 at 17:29

First of all get the path of the input directory inside your mapper with the help of FileSplit. Then append it to the name of the file which contains all these lines and read the lines of this file using FSDataInputStream. Something like this :

public void map(Object key, Text value, Context context)
                    throws IOException, InterruptedException {

        FileSplit fileSplit = (FileSplit)context.getInputSplit();
        FileSystem fs = FileSystem.get(context.getConfiguration());
        FSDataInputStream in = fs.open(new Path(fileSplit.getPath().getParent() + "/file.txt"));
        while(in.available() > 0){
                    FSDataOutputStream out = fs.create(new Path(in.readLine()));
        }
       //Proceed further....
}

FSDataInputStream is deprecated. you must use BufferedReader. (see more information in DataInputStream) — Yoonsung.Jung, Jul 26 '16 at 04:20

Hadoop Map Task : Read the content of a specified input file

2 Answers2