Locating Cloudera Manager HDFS config files

Question

I've installed a cluster via Cloudera Manager, and now I need to launch the cluster manually.

I've been using the following command:

$ sudo -u hdfs hadoop namenode / datanode / jobtracker

But then the dfs.name.dir is set up /tmp. I can't seem to find where cloudera manager has the HDFS config files. The ones in /usr/lib/hadoop-02*/conf seem to be minimal. They're missing the dfs.name.dir which is what I'm looking for particularly. I'm on an RHLE 6 system, by the way. Being lazy, I though I could just copy over cloudera manager's HDFS config files, so I don't have to manually create them, the copy them over to 6 nodes :)

Thanks

rational · Answer 1 · 2013-01-11T07:03:43.197

5

I was facing same problem. I was changing configuration parameters from cloudera manager ui but was clueless where my changes were getting updated on local file system.

I ran grep command and found out that in my case configuration were stored at /var/run/cloudera-scm-agent/process/*-hdfs-NAMENODE directory.

So David is right, whenever we change configs from ui and restart service, it creates new config. settings in /var/run/cloudera-scm-agent/process/ directory.

edited Jan 11 '13 at 07:03

answered Dec 28 '12 at 06:49

rational

143
10

1

Note that this is also nicely described in the [Cloudera blog](http://blog.cloudera.com/blog/2013/07/how-does-cloudera-manager-work/). – Robert Nov 15 '13 at 10:54

score 2 · Answer 2 · answered Oct 30 '12 at 18:22

Cloudera manager deploying config file each time you start cluster, each time in different directory. Directories are named after process id or something like this.
The configuration is passed explicitly to each deamon as parameter. So if you will look into command line of each hadoop deamons you can see where is configuration sitting (or just grep over disk for hdfs-site.xml. Names of config files are the same as usual.

score 2 · Accepted Answer · answered Jul 24 '14 at 17:08

Using CentOS 6.5, the Cloudera Manager special files do not show up in a SEARCH FILES result because their permissions are set to hide from all but the 'hdfs' user. In addition, there are multiple versions of hdfs-site.xml on the local drive some of which have partial amounts of real settings. The actual settings file is in the DATANODE folder not the NAMENODE folder as evidenced by the lack of dfs.datanode.data.dir values in the latter.

score 1 · Answer 4 · answered Nov 08 '13 at 17:23

I was in the same boat and found this answer:

To allow Hadoop client users to work with the HDFS, MapReduce, YARN and HBase services you created, Cloudera Manager generates client configuration files that contain the relevant configuration files with the settings from your services. These files are deployed automatically by Cloudera Manager based on the services you have installed, when you add a service, or when you add a Gateway role on a host.

You can download and distribute these client configuration files manually to the users of a service, if necessary.

The Client Configuration URLs command on the cluster Actions menu opens a pop-up that displays links to the client configuration zip files created for the services installed in your cluster. You can download these zip files by clicking the link.

See Deploying Client Configuration Files for more information on this topic.

On our system I got there via http://your_server:7180/cmf/services/status and clicked the Actions popup under the Add Cluster button. Hope that helps.

Locating Cloudera Manager HDFS config files

4 Answers4