How to access the static variable in the static inner class which extends reducer in mapreduce?

Question

Here is the code of main method of my program:

public class Path {
public static void main(String[] args) throws Exception {
    ArrayList<String> input = new ArrayList<String> ();
    input.add(args[0]);
    String output0="/output/path2";

    Route r1 =new Route(input,output0,2);
    r1.main(args);
    input.add(output0);
    String output1="/output/path3";

    Route r2 =new Route(input,output1,3);
    r2.main(args);
}}

Class route contains a static inner class extends mapper and a static inner class extends reducer. Here is part of the definition of class route which is relative to my question:

public class Route {
public static int len;
public static String output;
public static ArrayList<String> input = new ArrayList<String> ();

public static class RouteMapper extends Mapper<Object, Text, Text, Text> {
    public void map(Object key,Text value,Context context) throws IOException,InterruptedException {
       //do map
    }
}

public static class RouteReducer extends Reducer<Text, Text, Text, Text> {
    public void reduce(Text key, Iterable<Text> values, Context context) throws IOException,InterruptedException{
       //do reduce
    }
}

public Route(ArrayList<String> in,String out,int l){
    len = l;
    output = out;
    Iterator itr = in.iterator();
    while(itr.hasNext()){
        input.add(itr.next().toString());
    }
}

public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    //some configs of mapreduce
}}

As you can see in my main method, the value of static variable len should be 2 and 3 in two map/reduce phrases. However, when I try to write the value of len into context in reduce phrase, I got a value of zero, which is the default value of variable len.
Because of that, I cannot get the correct result as I wished to. I am quite confused why the inner static class RouteReducer cannot access the static variable of outer class Route.

This is an observation more than anything, but i think using static classes and static variables in MapReduce just confuses things. I guess the standard wordcount example uses them, but i wouldnt use that as a model for actually writing MR. Make you're Maps/Reducers their own (non-static) classes and pass config to them using the configuration, thats what its for. By default i wouldn't make anything static. — Binary Nerd, Feb 16 '17 at 13:42

score 2 · Accepted Answer · answered Feb 16 '17 at 13:49

Hadoop is distributed system. Compiled classes of mapper and reducer implementations are copied to other nodes where they work in their own JVMs. As such, main class and each of tasks see different instances of the same variables. You need to manually pass data from main class to your tasks.

Lightweight data may be passed from main class to map and reduce tasks by means of Configuration:

// in main
public static final String CONF_LEN = "conf.len";
...
conf.setInt(CONF_LEN, 3);

// in mapper
@Override
protected void setup(Mapper<Object, Text, Text, Text>.Context context) throws IOException, InterruptedException {
  super.setup(context);
  this.len = context.getConfiguration().getInt(CONF_LEN, -1);
}

Heavy data may be transferred by means of distributed cache.

How to access the static variable in the static inner class which extends reducer in mapreduce?

1 Answers1