Passing arguments to Hadoop mappers

Question

I'm using new Hadoop API and looking for a way to pass some parameters (few strings) to mappers.
How can I do that?

This solutions works for old API:

JobConf job = (JobConf)getConf();
job.set("NumberOfDocuments", args[0]);

Here, “NumberOfDocuments” is the name of parameter and its value is read from “args[0]“, a command line argument. Once you set this arguments, you can retrieve its value in reducer or mapper as follows:

private static Long N;
public void configure(JobConf job) {
     N = Long.parseLong(job.get("NumberOfDocuments"));
}

Note, the tricky part is that you cannot set parameters like this:

Configuration con = new Configuration();
con.set("NumberOfDocuments", args[0]);

both new and old methods are here: http://www.thecloudavenue.com/2011/11/passing-parameters-to-mappers-and.html — weefwefwqg3, Nov 21 '16 at 17:50

score 48 · Accepted Answer · edited Feb 08 '15 at 17:37

48

In the main method set the required parameter as below or using the -D command line option while running the job.

Configuration conf = new Configuration();
conf.set("test", "123");

Job job = new Job(conf);

In the mapper/reducer get the parameter as

Configuration conf = context.getConfiguration();
String param = conf.get("test");

edited Feb 08 '15 at 17:37

barik

160
2
12

answered Nov 23 '11 at 17:29

Praveen Sripati

32,799
16
80
117

and if we want to pass an integer parameter to the mapper? – MTT Dec 20 '13 at 22:04
VTT - the Configuration object has type-safe setters: setBoolean/setDouble/setEnum/etc... there are corresponding getters as well. – Robert Watkins Feb 04 '14 at 01:08
1

I am trying to set an array but i cant is there any other way i could do this? – tosi Dec 29 '14 at 16:45

Passing arguments to Hadoop mappers

1 Answers1

Linked