0

I'm trying to test my MapReduce with MRUnit, when I do the integration test, it works. I have some unit test that I want to pass them as well.

My MRUnit driver and MapReduce class are:

MapDriver<ImmutableBytesWritable, Result, ImmutableBytesWritable, KeyValue>

public final class HashMapper extends
    TableMapper<ImmutableBytesWritable, KeyValue>

When I define the input I get an error:

mapDriver.withInput(new ImmutableBytesWritable(Bytes
                .toBytes("query")), new Result(kvs1));

java.lang.NullPointerException
    at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:73)
    at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:91)
    at org.apache.hadoop.mrunit.internal.output.MockOutputCollector.collect(MockOutputCollector.java:48)
    at org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper$4.answer(AbstractMockContextWrapper.java:90)
    at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:29)
    at org.mockito.internal.MockHandler.handle(MockHandler.java:95)
    at org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47)

I guess that it's because it doesn't like the Result and KeyValue object since they're not Writable, but I don't undersntad why the integration tests works then. It was working before with Hbase 0.94 when all these objects implement Writable, now I'm working with HBase 0.96. Any clue how should I use MRUnit here?

Guille
  • 2,248
  • 3
  • 24
  • 42

1 Answers1

2

With the version HBase 0.96 some classes aren't implement Writable anymore, but people from HBase have created a new serialize classes for them.

So, the solutions is to indicate in the Configuration what classes MRUnit must use:

The property is called io.serializations The different serializes are:

Result class org.apache.hadoop.hbase.mapreduce.ResultSerialization KeyValue class org.apache.hadoop.hbase.mapreduce.KeyValueSerialization Put & Get classes org.apache.hadoop.hbase.mapreduce.MutationSerialization

Guille
  • 2,248
  • 3
  • 24
  • 42
  • +1 But I don't think there is serialization implemented for Put and Get. org.apache.hadoop.hbase.mapreduce.ResultSerialization implemnts Result class and org.apache.hadoop.hbase.mapreduce.KeyValueSerialization implements KeyValue – sethi Aug 11 '14 at 12:08
  • Ignore the above comment the formatting confused me. Later, I realized you mean to tell MutationSerialization implements serialization for Put/Get – sethi Aug 11 '14 at 12:16