Unit testing Mongo-Hadoop jobs with MrUnit

Question

How do you unit test mongo-hadoop jobs?

My attempt so far:

public class MapperTest {

    MapDriver<Object, BSONObject, Text, IntWritable> d;

    @Before
    public void setUp() throws IOException {
        WordMapper mapper = new WordMapper();
        d = MapDriver.newMapDriver(mapper);
    }

    @Test
    public void testMapper() throws IOException {

        BSONObject doc = new BasicBSONObject("sentence", "Two words");
        d.withInput(new Text("anykey"), doc );

        d.withOutput(new Text("Two"), new IntWritable(1));
        d.withOutput(new Text("words"), new IntWritable(1));

        d.runTest();
    }
}

Which produces this output:

No applicable class implementing Serialization in conf at io.serializations for class org.bson.BasicBSONObject

java.lang.IllegalStateException at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:67) at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:91) at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:104) at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:608) at org.apache.hadoop.mrunit.TestDriver.copyPair(TestDriver.java:612) at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:118) at org.apache.hadoop.mrunit.MapDriverBase.withInput(MapDriverBase.java:207) ...

Could you solve your issue? I am facing the same problem at the moment. I guess @Archit comment is not goig to work, since BSONObject is your input, not output — Blanca Hdez, Sep 24 '14 at 11:48
@Archit : counter documents in mongodb + amazon redshift together with some love from rabbitmq. so it is a mix. — marko, Dec 04 '14 at 22:17

score 1 · Answer 1 · answered Jan 31 '16 at 14:18

You need to set the serializer. Example : mapDriver.getConfiguration().setStrings("io.serializations", "org.apache.hadoop.io.serializer.WritableSerialization", MongoSerDe.class.getName());

MongoSerDe src: https://gist.github.com/lfrancke/01d1819a94f14da171e3

But I face error "org.bson.io.BasicOutputBuffer.pipe(Ljava/io/DataOutput;)I" while using this(MongoSerDe).

Unit testing Mongo-Hadoop jobs with MrUnit

1 Answers1