1

I am woking on HDFS and setting the replication factor to 1 in hfs-site.xml as follows:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>

  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/Users/***/Documnent/hDir/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/Users/***/Documnent/hDir/hdfs/datanode</value >
  </property>

  <property>
    <name>dfs.permissions</name>
    <value>false</value>
  </property>

</configuration>

But when i tried copying a file from local system to the hdfs file system, i found that the replication factor for that file was 3. Here is the code which is copying the file on hdfs:

    public class FileCopyWithWrite {

    public static void main(String[] args) {
        // TODO Auto-generated method stub

        String localSrc = "/Users/***/Documents/hContent/input/docs/1400-8.txt";
        String dst = "hdfs://localhost/books/1400-8.txt";
        try{

            InputStream in = new BufferedInputStream(new FileInputStream(localSrc));
            Configuration conf = new Configuration();;
            FileSystem fs = FileSystem.get(URI.create(dst), conf);

            OutputStream out = fs.create(new Path(dst), new Progressable() {

                public void progress() {
                    // TODO Auto-generated method stub
                    System.out.println(".");
                }
            });

            IOUtils.copyBytes(in, out, 4092, true);


        }catch(Exception e){
            e.printStackTrace();
        }
    }

}

Please find this screenshot showing the replication factor as 3:

enter image description here

Note that i have started in pseudo-distributed mode and i have updated the hdfs-site.xml according to the documentation in Hadoop The Definite Guide book. Any suggestion on why this is happening?

KayV
  • 12,987
  • 11
  • 98
  • 148
  • May I know how did you confirm that the file had a replication factor of 3? Also, did you restart the services after updating the replication factor or was the replication factor 1 from the start? – code May 04 '17 at 07:44
  • Was the replication factor 1 from the beginning or did you update it after starting the services? – code May 04 '17 at 07:58
  • I added the property dfs.replication with value as 1 and it is the same right from the beginning – KayV May 04 '17 at 07:59
  • I don't find any reason that hdfs would revert to its default replication factor. It has to be some misplaced config. Could you confirm the replication of this file via command line by running `hdfs dfs -ls`. The second column indicates the replication – code May 04 '17 at 08:41

2 Answers2

0

Deleting the namenode directory and datanode directory solved the issue for me. So I followed the following procedure:

  1. Stop the services, viz dfs, yarn and mr.

  2. Delete the namenode and datanote directories as specified in the hfs-site.xml file.

  3. Re-create the namenode and datanode directories.

  4. Restart the services, viz dfs, yarn and mr.

KayV
  • 12,987
  • 11
  • 98
  • 148
0

If above property was changed during the running state, then it requires the cluster to restart as the changed property gets reflected only after reloading them which is equivalent to restarting the cluster.

Julina
  • 709
  • 5
  • 13