0

I am getting this error while trying to run mrjob example on the hadoop cluster. I have set up my hadoop_home and I can also create a new dir on the hdfs file system. I can run python map-reduce if I use hadoop streaming. It's only with mrjob I am getting this issue.

When I run this command:

python mr_word_freq_count.py -r hadoop --hadoop-bin /usr/bin/hadoop -o hdfs:///user/zkdmkrq/out1 hdfs:///user/zkdmkrq/input1

I get:

no configs found; falling back on auto-configuration no configs found;
falling back on auto-configuration creating tmp directory
/tmp/mr_word_freq_count.zkdmkrq.20150226.172000.917957 writing wrapper
script to
/tmp/mr_word_freq_count.zkdmkrq.20150226.172000.917957/setup-wrapper.sh
STDERR: mkdir:
`hdfs:///user/zkdmkrq/tmp/mrjob/mr_word_freq_count.zkdmkrq.20150226.172000.917957/files/':
No such file or directory Traceback (most recent call last):   File
"mr_word_freq_count.py", line 37, in <module>
    MRWordFreqCount.run()   File "/usr/lib/python2.6/site-packages/mrjob/job.py", line 494, in run
    mr_job.execute()   File "/usr/lib/python2.6/site-packages/mrjob/job.py", line 512, in execute
    super(MRJob, self).execute()   File "/usr/lib/python2.6/site-packages/mrjob/launch.py", line 147, in
execute
    self.run_job()   File "/usr/lib/python2.6/site-packages/mrjob/launch.py", line 208, in
run_job
    runner.run()   File "/usr/lib/python2.6/site-packages/mrjob/runner.py", line 458, in run
    self._run()   File "/usr/lib/python2.6/site-packages/mrjob/hadoop.py", line 238, in _run
    self._upload_local_files_to_hdfs()   File "/usr/lib/python2.6/site-packages/mrjob/hadoop.py", line 265, in
_upload_local_files_to_hdfs
    self._mkdir_on_hdfs(self._upload_mgr.prefix)   File "/usr/lib/python2.6/site-packages/mrjob/hadoop.py", line 273, in
_mkdir_on_hdfs
    self.invoke_hadoop(['fs', '-mkdir', path])   File "/usr/lib/python2.6/site-packages/mrjob/fs/hadoop.py", line 109, in
invoke_hadoop
    raise CalledProcessError(proc.returncode, args) subprocess.CalledProcessError: Command '['/usr/bin/hadoop', 'fs',
'-mkdir',
'hdfs:///user/zkdmkrq/tmp/mrjob/mr_word_freq_count.zkdmkrq.20150226.172000.917957/files/']'
returned non-zero exit status 1
kichik
  • 33,220
  • 7
  • 94
  • 114
user1525721
  • 336
  • 5
  • 12

1 Answers1

0

I actually found the solution to this issue. I had to alter the mrjob/hadoop.py file. Here is the exact solution

https://github.com/Yelp/mrjob/issues/850

Hope it helps to anyone who encounters this issue.

user1525721
  • 336
  • 5
  • 12