I'm using Mrjob to run python code in Hadoop. I'm using a CDH package with virtual machine on a single node cluster. My mrjob ran correctly when I tested the code locally but when I ran on Hadoop cluster, it throw an error:
No module named mrjob
When I remove "sudo" command before python, I got following message.
no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
creating tmp directory /tmp/main_mrjob.cloudera.20131022.180113.820659
writing wrapper script to /tmp/main_mrjob.cloudera.20131022.180113.820659/setup-wrapper.sh
STDERR: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
STDERR: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
STDERR: at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
STDERR: at java.security.AccessController.doPrivileged(Native Method)
STDERR: at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
STDERR: at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
STDERR: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
STDERR: at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
STDERR: Could not find the main class: org.apache.hadoop.util.PlatformName. Program will exit.
STDERR: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FsShell
STDERR: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FsShell
STDERR: at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
STDERR: at java.security.AccessController.doPrivileged(Native Method)
STDERR: at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
STDERR: at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
STDERR: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
STDERR: at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
STDERR: Could not find the main class: org.apache.hadoop.fs.FsShell. Program will exit.
Traceback (most recent call last):
File "main_mrjob.py", line 17, in <module>
MRWordFrequencyCount.run()
File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/job.py", line 500, in run
mr_job.execute()
File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/job.py", line 518, in execute
super(MRJob, self).execute()
File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/launch.py", line 146, in execute
self.run_job()
File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/launch.py", line 207, in run_job
runner.run()
File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/runner.py", line 458, in run
self._run()
File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/hadoop.py", line 236, in _run
self._upload_local_files_to_hdfs()
File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/hadoop.py", line 263, in _upload_local_files_to_hdfs
self._mkdir_on_hdfs(self._upload_mgr.prefix)
File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/hadoop.py", line 271, in _mkdir_on_hdfs
self.invoke_hadoop(['fs', '-mkdir', path])
File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/fs/hadoop.py", line 104, in invoke_hadoop
raise CalledProcessError(proc.returncode, args)
subprocess.CalledProcessError: Command '['/usr/lib/hadoop-0.20-mapreduce/bin/hadoop', 'fs', '-mkdir', 'hdfs:///user/cloudera/tmp/mrjob/main_mrjob.cloudera.20131022.180113.820659/files/']' returned non-zero exit status 1
It seemns it can't "mkdir" on hdfs without sudo but with sudo it can't find mrjob. I'm really confused....
Thanks a lot!!
Yang
