2

I have a job whose output format is SequenceFileOuputFormat.

I set the output key and value class like this: 

conf.setOutputKeyClass(IntWritable.class);
conf.setOutputValueClass(SplitInfo.class);

The SplitInfo class implements Serializable,Writable

I set the io.serializations property as follows: 

conf.set("io.serializations","org.apache.hadoop.io.serializer.JavaSerialization," 
+ "org.apache.hadoop.io.serializer.WritableSerialization");

However, on the reducer side I get this error, telling me that Hadoop couldn't find a serializer: 

    java.lang.NullPointerException
at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:961)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:892)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:393)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:61)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:569)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:638)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)

Can anyone help, please ?

Razvan
  • 9,925
  • 6
  • 38
  • 51

2 Answers2

3

The problem was that I was making a stupid mistake: I was not updating a jar. So, basically SplitInfo was not implementing the Writable interface in the old (in use) jar.

As a general observation: the error specified in the OP has as underlying cause the fact that HADOOP can't find a Serializer for a specific type which you're trying to serialize (being directly or indirectly, e.g. by using that type as an output key/value). Hadoop cannot find a Serilizer for one of the 2 reasons:

  1. your type is not serializable (i.e. it doesn't implement Writable or Serializable)
  2. There is no Serializer available to Hadoop for the type of serialization your type implements (e.g.: your type implements Writable but hadoop for one reason or another cannot use the org.apache.hadoop.io.serializer.WritableSerialization class)
Razvan
  • 9,925
  • 6
  • 38
  • 51
0

I think you're trying to do something you don't need to. Your output value only needs to implement the Writable interface and you should just set the output format.

conf.setOutputFormatClass(SequenceFileOutputFormat.class);

You only use the "io.serializations" configuration if you want to use a different serialization framework, which it doesn't look like you need.

Binary Nerd
  • 13,872
  • 4
  • 42
  • 44
  • I also use DefaultStringifier which seems to need the io.serialization set. Otherwise it doesn't work! – Razvan Sep 16 '12 at 08:49
  • Im sorry i'm not following. You shouldn't need anything related to the io.serializations stuff to write a custom Writable value object to a SequenceFileOutputFormat. Why are you using the DefaultStringifier? – Binary Nerd Sep 16 '12 at 15:01
  • DefaultStringifier to store something in the conf file – Razvan Sep 16 '12 at 17:49