Hadoop Mapreduce Error Input path does not exist: hdfs://localhost:54310/user/hduser/input"

Question

I have installed hadoop 2.6 in Ubuntu Linux 15.04 and its running fine. But, when I am running a sample test mapreduce program, its giving the following error:

org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://localhost:54310/user/hduser/input.

Kindly help me. Below is the complete details of the error.

hduser@krishadoop:/usr/local/hadoop/sbin$ hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount input output
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
15/08/24 15:22:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/08/24 15:22:38 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
15/08/24 15:22:38 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15/08/24 15:22:39 INFO mapreduce.JobSubmitter: Cleaning up the staging area file:/app/hadoop/tmp/mapred/staging/hduser1122930879/.staging/job_local1122930879_0001
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://localhost:54310/user/hduser/input
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:321)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:264)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:385)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:597)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

[![the tutorial from tutorialspoint](https://i.stack.imgur.com/hJE2b.png)](https://i.stack.imgur.com/hJE2b.png) View more from [this](https://www.tutorialspoint.com/hadoop/hadoop_command_reference.htm) , Or you can seek the terminal from [The Official Guides](https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html) — Benchur Wong, May 06 '19 at 08:04

Amal G Jose · Answer 1 · 2015-08-24T10:31:02.990

9

Seems like you mentioned a wrong input path. Hadoop is searching for an input path at /user/hduser/input. Hadoop also follows unix like tree structure. If you simply mention a directory input it will be taken as /user/{username}/input.

hadoop fs -mkdir -p /user/hduser/input

hadoop fs -put <datafile>  /user/hduser/input

edited Aug 24 '15 at 10:31

answered Aug 24 '15 at 10:23

Amal G Jose

2,486
1
20
35

what should be the ? – Sami Jul 11 '16 at 09:14
is the file that you want to load into hdfs – Amal G Jose May 21 '18 at 07:01
Absolute path name of the file. – yunus Mar 28 '19 at 01:09

Supun Wijerathne · Answer 2 · 2017-03-03T00:41:32.417

5

If you see this path (file) physically and still getting the error, you may have confused with local file system and Hadoop Distributed File System(HDFS). In order to run this map-reduce, this file should be located in HDFS (locating only inside local file system will not do it.).

You can import local file system files into HDFS by this command.

hadoop fs -put <local_file_path> <HDFS_diresctory>

You confirm that the file that you imported exists in HDFS by this command.

hadoop fs -ls <HDFS_path>

edited Mar 03 '17 at 00:41

answered May 02 '16 at 11:31

Supun Wijerathne

11,964
10
61
87

1

This is used command when HDFS model. grunt> pig -x mapreduce – Shyam Gupta Mar 02 '17 at 17:17

score 1 · Answer 3 · answered Aug 24 '15 at 21:05

You must create and upload your input before executing your hadoop job. For example, if you need to upload input.txt file, you should do the following:

$HADOOP_HOME/bin/hdfs dfs -mkdir /user/hduser/input
$HADOOP_HOME/bin/hdfs dfs -copyFromLocal $HADOOP_HOME/input.txt /user/hduser/input/input.txt

The first line creates the directory, and the other upload your input file into hdfs (hadoop fylesystem).

score 1 · Answer 4 · answered Aug 25 '15 at 04:53

When you compile any jar file using input and output file/directory, you should make sure that the input file is already created(in the specified path) and output file does not exist.

If you want to give a text file as input file, first copy a text file from local file system to hdfs and compiling it by using the following commands

hadoop fs -copyFromLocal /input.txt /user/hduser/input.txt
/usr/local/hadoop/sbin$ yarn jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount /user/hduser/input.txt /output

/input.txt may be replaced with address of any text file.

score 1 · Answer 5 · answered Feb 06 '17 at 14:15

1

You need to start Pig in local mode and not cluster node:

pig -x local

answered Feb 06 '17 at 14:15

Walker Rowe

953
1
12
24

This is used when we are using localmode. if we not run this command, then it will give the error file not found... – Shyam Gupta Mar 02 '17 at 17:18

score 0 · Answer 6 · edited Apr 15 '17 at 18:29

Program is not able to find the Hadoop path for the inputs. It is searching in the local system files rather than Hadoop's DFS.

This problem will go away when your program is able to locate the HDFS location. We need to let the program understand the HDFS location given in the configuration file. To do that, add these lines in your program code.

Configuration conf = new Configuration();

conf.addResource(new Path("/usr/local/hadoop/hadoop-2.7.3/etc/hadoop/core-site.xml"));
conf.addResource(new Path("/usr/local/hadoop/hadoop-2.7.3/etc/hadoop/hdfs-site.xml"));

score 0 · Answer 7 · edited Apr 04 '19 at 08:36

0

you should make the directory in HDFS: for instance, "hadoop fs -mkdir /input_dir"
Then when you run your MapReduce program. You should mention the absolute path of input directory, so the format should be:
hadoop jar jarFileName.jar className /input_dir /outputdir right

The following is wrong because it is relative path hadoop jar jarFileName.jar className input_dir outputdir wrong

edited Apr 04 '19 at 08:36

Giulio Caccin

2,962
6
36
57

answered Apr 04 '19 at 05:37

Zhihui Shao

377
3
5

user1900344 · Answer 8 · 2019-07-08T05:51:04.687

-2

If you find /bin/bash: /bin/java: No such file or directory in log, try setting JAVA_HOME in /etc/hadoop/hadoop-env.sh

edited Jul 08 '19 at 05:51

answered Apr 20 '16 at 09:18

user1900344

259
3
12

Hadoop Mapreduce Error Input path does not exist: hdfs://localhost:54310/user/hduser/input"

8 Answers8