I am trying to pass a complex writable between mapper and reducer, more specifically ArrayWritable of ObjectWritables.
public class ObjectArrayWritable extends ArrayWritable {
public ObjectArrayWritable() {
super(ObjectWritable.class);
}
}
My mapper: Mapper<LongWritable, Text, Text, ObjectArrayWritable>
My reducer: Reducer<Text,ObjectArrayWritable,Text,ObjectArrayWritable>
The reducer crashes on the readfields() method of the writable interface and throws the following exception:
java.lang.NullPointerException
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:183)
at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
at org.apache.hadoop.io.ArrayWritable.readFields(ArrayWritable.java:90)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
at org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
When i tried to implement my own custom writable and debugged it i noticed that the buffer of the DataInput in the readfields(DataInput data) is received null..
Note that my error is similar to: hadoop + Writable interface + readFields throws an exception in reducer but the cleaning of the object didn't help..
Can anyone assist?
