hadoop partitioner getting incorrect reduce count

Question

I'm working on partitioner today. Its the basic program in hadoop custom partitioners. Below is my partitioner code snippet.

public class VowelConsPartitioner extends Partitioner {

@Override
public int getPartition(Text letterType, IntWritable count, int redCnt) {
    // TODO Auto-generated method stub

    //System.out.println("reduce cnt in partitioner: "+redCnt);
    if(letterType.toString().equalsIgnoreCase("vowel")){

        //System.out.println("vowel sound: "+1%redCnt);
        return letterType.toString().hasCode()%redCnt;
    }

    if(letterType.toString().equalsIgnoreCase("consonent")){
        //System.out.println("Cons sound: "+2%redCnt);
        return letterType.toString().hasCode()%redCnt;
    }

    else
        return 0;
}

}

And I set my reducers in my driver class like this....

job.setNumReduceTasks(3); job.setPartitionerClass(VowelConsPartitioner.class);

I want to keep more than 1 reducer. But I'm getting the o/p in one reducer only. Moreover if you see the partitioner code, the first sysout(I've commented) was giving me redCnt as 1. I'm not sure how thats happening when I set its count to 3 from my driver class. Could someone help me out on this?

FYI... I'm making jar & running this on HDFS.

All, its hashCode(). Not hasCode() my bad!!! its a typo – Santosh Batta Aug 25 '13 at 05:32 — Santosh Batta, Aug 25 '13 at 05:32

score 0 · Answer 1 · answered Aug 27 '13 at 05:39

0

Your Logic is seems to be correct! I guess u need to create a jar file and run it in terminal to get the partitioned result.

Cheers!

answered Aug 27 '13 at 05:39

Krishna Kalyan

1,672
2
20
43

Logic Partitioner https://gist.github.com/krishnakalyan3/6349973 Driver https://gist.github.com/krishnakalyan3/6349965 Reducer https://gist.github.com/krishnakalyan3/6349961 Mapper https://gist.github.com/krishnakalyan3/6349950 – Krishna Kalyan Aug 27 '13 at 05:42

hadoop partitioner getting incorrect reduce count

1 Answers1