Apache MRUnit is a Java library that helps developers unit test Apache Hadoop map reduce jobs.
Apache MRUnit is a Java library that helps developers unit test Apache Hadoop map reduce jobs.
Questions tagged [mrunit]
64 questions
0
votes
1 answer
Testing for a specific exception in MRUnit
In TestNG (or JUnit), it's simple. Goes something like:
@Test(expectedExceptions = NullPointerException)
public void test() throws NullPointerException {
String x = null;
String y = "y";
Assert.assertEquals(x.someMethod(), y);
}
Above…
danksim
- 617
- 3
- 9
- 27
0
votes
1 answer
MRUnit test case for Driver
I have written MRunit with following code:
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "file:///");
conf.set("fs.default.name", "file:///");
conf.set("mapreduce.framework.name",…
dinesh028
- 2,137
- 5
- 30
- 47
0
votes
1 answer
How do I run MRUnit tests using mvn test command like JUnit tests
I am able to run the MRUnit tests from eclipse by right clicking the project and clicking Run as JUnit test, but when I run the same MRUnit test program using the mvn test command on the command line, I get the following output with no…
Abu Sulaiman
- 1,477
- 2
- 18
- 32
0
votes
0 answers
Treeset is not sorting values in hadoop mapper map function
MAPPER CLASS
import java.io.IOException;
import java.util.TreeSet;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class…
selvam raman
- 9
- 3
0
votes
1 answer
How to test hadoop mapreduce with hdfs?
I am using MRUnit to write unit tests for my mapreduce jobs.
However, I am having trouble including hdfs into that mix. My MR job needs a file from hdfs. How do I mock out the hdfs part in MRUnit test case?
Edit:
I know that I can specify…
feroze
- 7,380
- 7
- 40
- 57
0
votes
1 answer
How to test reducer with avro params in MRUnit?
I'm trying to test the Reducer bellow
Reducer, AvroKey, NullWritable>
For these purpose I use the next code in the test:
CustomReducer reducer = new CustomReducer();
reduceDriver = new…
Speise
- 789
- 1
- 12
- 28
0
votes
2 answers
MRUnit does not work with MultipleOutputs
When I run a basic MRUnit with MultipleOutputs I get the following exception:
java.lang.NullPointerException
at org.apache.hadoop.fs.Path.(Path.java:105)
at org.apache.hadoop.fs.Path.(Path.java:94)
at…
Horatiu Jeflea
- 7,256
- 6
- 38
- 67
0
votes
1 answer
Haddop MRUnit MapDriver.addInput() giving NotSerializableException: java.nio.HeapByteBuffer error
I am using MRunit to unit test our MapReducer below are pom changes
org.apache.mrunit
mrunit
1.1.0
hadoop1
…
mightymahesh
- 303
- 3
- 14
0
votes
1 answer
MRUnit Maven project import issue
I am studying Hadoop on the book "Hadoop, the definitive guide 4th edition" by O'Really and I am having an issue in trying to create a Maven project for MRUnit testing. I just created the pom.xml file as shown in the book (just changing my hadoop…
mgaido
- 2,987
- 3
- 17
- 39
0
votes
0 answers
Chaining MapReduce jobs in unit tests (mrunit 1.1.0)
I have a mapreduce process that has 2 mapper/reducer pairs chained together.
How can I go about unit-testing the whole chain?
I've looked into the MapReduceDriver and can't seem to figure out how to pipe the output of the first job into the…
mfau
- 11
- 4
0
votes
1 answer
Hadoop MRUnit IllegalStateException when using Hadoop-The Definitive Guide code
I am studying Hadoop from the Definitive Guide book and tried to execute this piece of code which is resulting in the error.
Example from Chapter-5. Link to Github…
BrownBeast
- 21
- 1
- 9
0
votes
1 answer
Testing multiple outputs with MRUnit 1.1.0
Related question @ Testing multiple outputs with MRUnit but answer is not applicable for the newer version 1.1.0
The question is how to setup the multiple named outputs so that the underlying mock implementations recognize the named paths. I am…
Irfan
- 71
- 1
- 4
0
votes
2 answers
MRUnit passing values in hbase Result object
I am testing my mapper with MRUnit. I am passing key and list of values as input to the mapper from the test class.
The problem is :
String key=1234_abc;
ArrayList list = new ArrayList();
KeyValue k1 = new…
user2401464
- 517
- 2
- 8
- 20
0
votes
1 answer
Avro with MRUnit gives InstantiationException
I'm using:
hadoop-client 2.2.0
mrunit 1.0.0
avro 1.7.6
avro-mrunit 1.7.6
... and the entire thing is being built and tested using Maven.
I was getting a NullPointerException until I followed the instructions at MRUnit with Avro…
seawolf
- 2,147
- 3
- 20
- 37
0
votes
1 answer
MRUnit with HBase and MapReduce gets an error with serialization
I'm trying to test my MapReduce with MRUnit, when I do the integration test, it works. I have some unit test that I want to pass them as well.
My MRUnit driver and MapReduce class are:
MapDriver
Guille
- 2,248
- 3
- 24
- 42