Can't run a giraph SimpleInDegreeCountComputation

Question

I'm trying to run the SimpleInDegreeCountComputation example included with Giraph. My approach is as follows:

SimpleInDegreeCountComputation.java:

    public class SimpleInDegreeCountComputation extends BasicComputation
              <LongWritable, LongWritable, DoubleWritable, DoubleWritable> {
    .......

I then try running it like so:

    hadoop jar /path-to-giraph-folder/giraph-examples/target/giraph-examples-1.1.0- 
    SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar 
    org.apache.giraph.GiraphRunner  
    org.apache.giraph.examples.SimpleInDegreeCountComputation 
    -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat 
    -vip /path-to-input-file 
    -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat 
    -op /path-to-output-file -w 1

The result is the following:

    14/05/18 18:58:40 INFO utils.ConfigurationUtils: No edge input format specified. 
    Ensure your InputFormat does not require one.
    14/05/18 18:58:40 INFO utils.ConfigurationUtils: No edge output format specified.  
    Ensure your OutputFormat does not require one.
    Exception in thread "main" java.lang.IllegalArgumentException: checkClassTypes: vertex  
    value types not assignable, computation - class org.apache.hadoop.io.LongWritable,   
    VertexInputFormat - class org.apache.hadoop.io.DoubleWritable
at org.apache.giraph.job.GiraphConfigurationValidator.checkAssignable(GiraphConfigurationValidator.java:381)
at org.apache.giraph.job.GiraphConfigurationValidator.verifyVertexInputFormatGenericTypes(GiraphConfigurationValidator.java:228)
at org.apache.giraph.job.GiraphConfigurationValidator.validateConfiguration(GiraphConfigurationValidator.java:141)
at org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:214)
at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

I'm not quite sure what I'm doing wrong. If anyone can point me in the right direction, or link to a resource that explains an easier way of what I'm trying to do, I'd greatly appreciate it! I think that the problem may be the wrong formats (-vif). The input file I'm using is the following:

    [0,0,[[1,5],[2,9]]]
    [1,0,[[0,5],[3,3]]]
    [2,0,[[0,9],[3,3],[4,3]]]
    [3,0,[[1,3],[2,3],[4,2]]]
    [4,0,[[2,3],[3,3]]]

score 6 · Accepted Answer · answered May 18 '14 at 19:20

Looking at the definition of the compute and vertex-input classes, it seems that the JsonLongDoubleFloatDoubleVertexInputFormat isn't compatible with SimpleInDegreeCountComputation

SimpleInDegreeCountComputation:

public class SimpleInDegreeCountComputation extends BasicComputation<
    LongWritable, LongWritable, DoubleWritable, DoubleWritable> {

BasicComputation:

/**
 * Computation in which both incoming and outgoing message types are the same.
 *
 * @param <I> Vertex id
 * @param <V> Vertex data
 * @param <E> Edge data
 * @param <M> Message type
 */
public abstract class BasicComputation<I extends WritableComparable,
    V extends Writable, E extends Writable, M extends Writable>
    extends AbstractComputation<I, V, E, M, M> {
}

You can see that:

Vertex id is of Type LongWritable
Vertex data is of Type LongWritable
Edge data is of Type DoubleWritable

... on the other hand the InputFormat you are trying to use ...

JsonLongDoubleFloatDoubleVertexInputFormat:

public class JsonLongDoubleFloatDoubleVertexInputFormat extends
    TextVertexInputFormat<LongWritable, DoubleWritable, FloatWritable> {

TextVertexInputFormat:

/**
 * Abstract class that users should subclass to use their own text based
 * vertex input format.
 *
 * @param <I> Vertex index value
 * @param <V> Vertex value
 * @param <E> Edge value
 */
@SuppressWarnings("rawtypes")
public abstract class TextVertexInputFormat<I extends WritableComparable,
    V extends Writable, E extends Writable>
    extends VertexInputFormat<I, V, E> {

You can see that:

Vertex id is of Type LongWritable
Vertex data is of Type DoubleWritable
Edge data is of Type FloatWritable

Since it's LongWritable, DoubleWritable and FloatWritable and not Long, Double and Float - those types cannot be converted automatically.

I couldn't find any InputFormat you could use instead, so you'll either need to modify the existing JsonLongDoubleFloatDoubleVertexInputFormat or modify the algorithm to use NullWritable for the Edge data type. I don't see anywhere the Edge-data to be used so it can as well be null. In this case, you can use LongLongNullTextInputFormat.

Hi zahorak! Thank you for the answer. I tried to do so, but now the problem is to create a new jar that contains a modified file. I'm not sure how to do that. I unpacked giraph-examples-1.1.0- SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar to get a look how it is made. But i'm not still able no create a correct jar file because I don't have all files that are necessary (that are contained in giraph-examples-1.1.0- SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar ). Any suggestions? — user3489477, May 20 '14 at 18:37
all my code which I wrote in giraph, I just did inside the giraph-examples. Just check out from github https://github.com/apache/giraph and edit it, afterwards run `mvn clean install -DskipTests` and you should get all the jars you need. https://github.com/pgrm/giraph is my repository with my changes to giraph-examples, to be able to run my code. — peter, May 21 '14 at 12:39
Thank you zahorak. So according to your suggestions my solution was: 1) get the code from github 2) modify the file SimpleInDegreeCountComputation 3) create the jars with mvn install Now it works :)) — user3489477, May 22 '14 at 19:57
Step 4) could be to send a pull request back, so that others wouldn't have this issue anymore (but I'm not sure if they'll accept it) — peter, May 22 '14 at 22:14

Can't run a giraph SimpleInDegreeCountComputation

1 Answers1