38

I am running hadoop with default configuration with one-node cluster, and would like to find where HDFS stores files locally.

Any ideas?

Thanks.

crypto5
  • 851
  • 2
  • 10
  • 16

6 Answers6

31

You need to look in your hdfs-default.xml configuration file for the dfs.data.dir setting. The default setting is: ${hadoop.tmp.dir}/dfs/data and note that the ${hadoop.tmp.dir} is actually in core-default.xml described here.

The configuration options are described here. The description for this setting is:

Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored.

user2966533
  • 3
  • 2
Binary Nerd
  • 13,872
  • 4
  • 42
  • 44
15

Seems like for the current version(2.7.1) the dir is 

/tmp/hadoop-${user.name}/dfs/data

Based on dfs.datanode.data.dir, hadoop.tmp.dir setting from: http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/core-default.xml

MaxNevermind
  • 2,784
  • 2
  • 25
  • 31
14

As "more recent answer" and to clarify hadoop version numbers:

If you use Hadoop 1.2.1 (or something similar), @Binary Nerd's answer is still true.

But if you use Hadoop 2.1.0-beta (or something similar), you should read the configuration documentation here and the option you want to set is: dfs.datanode.data.dir

contradictioned
  • 1,253
  • 2
  • 14
  • 26
0

For hadoop 3.0.0, the hdfs root path is as given by the property "dfs.datanode.data.dir"

Binita Bharati
  • 5,239
  • 1
  • 43
  • 24
0

First find the Hadoop directory present in /usr/lib. There you can find the etc/hadoop directory, where all the configuration files are present.

In that directory you can find the hdfs-site.xml file which contains all the details about HDFS. There you find 2 properties:

dfs.namenode.name.dir – tells where the namenode stores the metadata on the local filesystem.

dfs.datanode.data.dir – tells where the datanode stores the data on the local filesystem

Viren Patel
  • 1
-4

Run this in the cmd prompt, and you will get the HDFS location:

bin/hadoop fs -ls /
dilshad
  • 734
  • 1
  • 10
  • 27
  • As presented, this command just gives an error. And you fixed the command, it wouldn't answer the original question at all, -ls isn't going to help in this regard. – derekv Oct 08 '15 at 16:34