2

Using hadoop multinode setup (1 mater , 1 salve)

After starting up start-mapred.sh on master , i found below error in TT logs (Slave an)

org.apache.hadoop.mapred.TaskTracker: Failed to get system directory

can some one help me to know what can be done to avoid this error

I am using Hadoop 1.2.0 jetty-6.1.26 java version "1.6.0_23"

mapred-site.xml file 

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>master:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
<property>
  <name>mapred.map.tasks</name>
    <value>1</value>
      <description>
          define mapred.map tasks to be number of slave hosts
</description>
</property>

<property>
  <name>mapred.reduce.tasks</name>
    <value>1</value>
      <description>
          define mapred.reduce tasks to be number of slave hosts
</description>
</property>

</configuration>

core-site.xml

<configuration>
<property>
  <name>fs.default.name</name>
  <value>hdfs://master:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/home/hduser/workspace</value>
</property>
</configuration>
Surya
  • 3,408
  • 5
  • 27
  • 35

2 Answers2

2

It seems that you just added hadoop.tmp.dir and started the job. You need to restart the Hadoop daemons after adding any property to the configuration files. You have specified in your comment that you added this property at a later stage. This means that all the data and metadata along with other temporary files is still in the /tmp directory. Copy all those things from there into your /home/hduser/workspace directory, restart Hadoop and re run the job.

Do let me know the result. Thank you.

Tariq
  • 34,076
  • 8
  • 57
  • 79
  • Re-format filesystem in /tmp/hadoop-root/dfs/name ? (Y or N) y Format aborted in /tmp/hadoop-root/dfs/name ; issue on format , can u advise @Tariq – Surya Jun 19 '14 at 06:11
  • this woked fine after removing temp directory – Surya Jun 19 '14 at 06:40
  • but issue with connceting to name node , org.apache.hadoop.mapred.JobTracker: Problem connecting to HDFS Namenode... re-trying @ Tariq – Surya Jun 19 '14 at 06:41
0

If, it is your windows PC and you are using cygwin to run Hadoop. Then task tracker will not work.

Sandeep Singh
  • 7,790
  • 4
  • 43
  • 68
  • @Tariq-- I didn't get your question. Please explain what you want to say :) – Sandeep Singh Jul 29 '13 at 10:32
  • 1
    I would like to know that how did you arrive at the conclusion that TT will not run if you are using Hadoop through cygwin. Thank you. – Tariq Jul 29 '13 at 11:21
  • @Tariq--This is my personal experience, When I was installing Hadoop on my Windows PC. All Job got started successfully. but, while running the MR program. I got error while checking tasktracker log **Tasktracker currently supporting linux machine only**. and My job got fail :) Please check these link as well **http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml** and **http://hadoop.apache.org/docs/stable/cluster_setup.html** – Sandeep Singh Jul 29 '13 at 12:13
  • 1
    Thank you for the pointers. You can do it, IMHO. Process is not smooth though. – Tariq Jul 29 '13 at 20:39