Mahout: Missing class to create Sequence Files

Question

I'm following the instructions at the mahout site for converting an existing file to a sequence file:

VectorWriter vectorWriter = SequenceFile.createWriter(filesystem,
                                                  configuration,
                                                  outfile,
                                                  LongWritable.class,
                                                  SparseVector.class);

long numDocs = vectorWriter.write(new VectorIterable(), Long.MAX_VALUE);

I've included the mahout jar in my maven project:

    <dependency>
        <groupId>org.apache.mahout</groupId>
        <artifactId>mahout-core</artifactId>
        <version>0.9</version>
    </dependency>

But it won't write the file.

I get this error:

Caused by: java.lang.NullPointerException
at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:963)
at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.<init>(SequenceFile.java:1136)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:397)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:284)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:265)

which, after further investigation, is caused by:

Serilization class not found: java.lang.ClassNotFoundException: org.apache.hadoop.io.serializer.WritableSerialization

That suggests to me that I'm missing a jar-- does anyone know which one?

score 0 · Answer 1 · answered Mar 11 '15 at 14:33

The problem was that I was using it within Lenskit, and the Configuration class tries to use Thread.currentThread().getContextClassLoader() which (for whatever reason) doesn't have the mahout or hadoop packages. The complete code is:

final Configuration configuration = new Configuration();
configuration.setClassLoader(Configuration.class.getClassLoader());

final Path path = new Path(POINTS_PATH + "/pointsFile");

LocalFileSystem fs = new LocalFileSystem();
fs.initialize(URI.create(POINTS_PATH + "/pointsFile"), configuration);

final SequenceFile.Writer writer =
        SequenceFile.createWriter(
                fs,
                configuration,
                path,
                Text.class,
                VectorWritable.class);

Mahout: Missing class to create Sequence Files

1 Answers1