Custom Partitioner Error

Question

I am writing my own custom Partitioner(Old Api) below is the code where I am extending Partitioner class:

public static class WordPairPartitioner extends Partitioner<WordPair,IntWritable> {

   @Override
   public int getPartition(WordPair wordPair, IntWritable intWritable, int numPartitions) {
        return wordPair.getWord().hashCode() % numPartitions;
    }
}

Setting the JobConf:

conf.setPartitionerClass(WordPairPartitioner.class);

WordPair Class contains:
private Text word;
private Text neighbor;

Questions:
1. I am getting error:"actual argument class (WordPairPartitioner) cannot convert to Class (?extends Partitioner).
2. Is this a right way to write the custom partitioner or do I need to override some other functionality as well?

Amar · Accepted Answer · 2013-03-30T12:29:49.387

3

I believe you are mixing up old API(classes from org.apache.hadoop.mapred.*) and new API(classes from org.apache.hadoop.mapreduce.*)

Using old API, you may do as follows:

import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.Partitioner;
public static class WordPairPartitioner implements Partitioner<WordPair,IntWritable> {

   @Override
   public int getPartition(WordPair wordPair, IntWritable intWritable, int numPartitions) {
        return wordPair.getWord().hashCode() % numPartitions;
    }


   @Override
   public void configure(JobConf arg0) {

   }
}

edited Mar 30 '13 at 12:29

answered Mar 30 '13 at 12:24

Amar

11,930
5
50
73

Thanks man! I don't know how come I forgot to modify the headers..thanks for concrete answer...I mixed old and new api''s ..:) – JackSparrow Mar 30 '13 at 18:15

score 2 · Answer 2 · answered Mar 30 '13 at 14:16

2

In addition to Amar's answer, you should handle the eventuality of hashCode returning a negative number by bit masking:

@Override
public int getPartition(WordPair wordPair, IntWritable intWritable, int numPartitions) {
    return (wordPair.getWord().hashCode() % numPartitions) & 0x7FFFFFFF;
}

answered Mar 30 '13 at 14:16

Chris White

29,949
4
71
93

Thanks Chris...sure I'll take care of this. – JackSparrow Mar 30 '13 at 18:15

Custom Partitioner Error

2 Answers2