I am writing MRjob and want to partition my reducer output on key based. And I am using these options and get following error.How to use keyfieldbasedpartitioner? Do I need download something for this. And MRJOB is written in python.
Step 1 of 1 failed: Command '['hadoop', 'jar',
'/usr/lib/hadoop-mapreduce/hadoop-streaming.jar', '-files',
'hdfs://hdpb-dfs/tmp/20170716.162009.525122/files/abc.py#parsec_status_error_fedactivity.py,hdfs://hdpb-dfs/tmp/20170716.162009.525122/files/setup-wrapper.sh#setup-wrapper.sh',
'-archives',
'hdfs://hdpb-dfs/tmp/20170716.162009.525122/files/mrjob.tar.gz#mrjob.tar.gz',
'-D', 'mapreduce.job.name=abc', '-D', 'mapreduce.job.reduces=2',
'-D', 'mapreduce.job.split.metainfo.maxsize=-1', '-D',
'mapreduce.map.failures.maxpercent=1', '-D',
'mapreduce.map.java.opts=-Xmx1g', '-D',
'mapreduce.map.memory.mb=2048', '-D',
'mapreduce.output.fileoutputformat.compress=true', '-D',
'mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec',
'-D', 'mapreduce.partition.keypartitioner.options=-k1', '-D',
'mapreduce.reduce.java.opts=-Xmx2g', '-D',
'mapreduce.reduce.memory.mb=3072', '-D',
'mapreduce.reduce.shuffle.input.buffer.percent=0.4', '-D',
'mapreduce.reduce.shuffle.merge.percent=0.4', '-D',
'stream.map.input.ignoreKey=true', '-D',
'stream.num.map.output.key.fields=5', '-libjars',
'/opt/parsec/lib/correctionlayer2.jar', '-partitioner',
'org.apache.Hadoop.mapred.lib.KeyFieldBasedPartitioner', '-input',
'hdfs:////10.134.71.100.1500076800077.gz', '-output',
'hdfs:///20170715', '-mapper', 'sh -ex setup-wrapper.sh python abc.py
--step-num=0 --mapper', '-reducer', 'sh -ex setup-wrapper.sh python abc.py --step-num=0 --reducer']' returned non-zero exit status 256de
here
Running step 1 of 1...
-partitioner : class not found : org.apache.Hadoop.mapred.lib.KeyFieldBasedPartitioner
Try -help for more information
Streaming Command Failed!
