How to append the data to a file which is stored in HDFS

Question

I want to store the details of current running queries, like file Names on which the queries are executing and time. for this i created one file in HDFS, tried to writing the information. But the problem is how to append the data to the existing file. Please help me. Thanks in Advance

Answer 1

First of all stop all the Hadoop daemons and add the following property in your hdfs-site.xml :

<property>
       <name>dfs.support.append</name>
       <value>true</value>
</property> 

Now, restart the daemons and try this code :

public class HDFSAppend {

    public static void main(String[] args) throws IOException {
        // TODO Auto-generated method stub
        Configuration conf = new Configuration();
        conf.addResource(new Path("/path/to/your/hadoop/directory/conf/core-site.xml"));
        conf.addResource(new Path("/path/to/your/hadoop/directory/conf/hdfs-site.xml"));
        FileSystem fs = FileSystem.get(conf);
        FSDataOutputStream out = fs.append(new Path("/demo.txt"));
        out.writeUTF("Append demo...");
        fs.close();

    }    
}

HTH

Answer 2

You can do it from the command line:

$ hadoop fs -appendToFile <local_file> <hdfs_file>

If you are using java see this question: Write a file in hdfs with Java

Answer 3

The simplest way of doing it from command line would be:

echo -e 'Hello\nWorld' | hadoop dfs -put - /1.txt

Which will create a file 1.txt in the path / of HDFS and will store two lines Hello\nWorld in it.

Answer 4

Use command : hdfs dfs -put file_location hdfs_location

Heading

Note: hdfs_location is enable

Answer 5

Solved..!!

Append is supported in HDFS.

You just have to do some configurations and simple code as shown below :

Step 1: set dfs.support.append as true in hdfs-site.xml :

<property>
   <name>dfs.support.append</name>
   <value>true</value>
</property>

Stop all your daemon services using stop-all.sh and restart it again using start-all.sh

Step 2 (Optional): Only If you have a singlenode cluster , so you have to set replication factor to 1 as below :

Through command line :

./hdfs dfs -setrep -R 1 filepath/directory

Or you can do the same at run time through java code:

fShell.setrepr((short) 1, filePath);  

Step 3 : Code for Creating/appending data into the file :

public void createAppendHDFS() throws IOException {
    Configuration hadoopConfig = new Configuration();
    hadoopConfig.set("fs.defaultFS", hdfsuri);
    FileSystem fileSystem = FileSystem.get(hadoopConfig);
    String filePath = "/test/doc.txt";
    Path hdfsPath = new Path(filePath);
    fShell.setrepr((short) 1, filePath); 
    FSDataOutputStream fileOutputStream = null;
    try {
        if (fileSystem.exists(hdfsPath)) {
            fileOutputStream = fileSystem.append(hdfsPath);
            fileOutputStream.writeBytes("appending into file. \n");
        } else {
            fileOutputStream = fileSystem.create(hdfsPath);
            fileOutputStream.writeBytes("creating and writing into file\n");
        }
    } finally {
        if (fileSystem != null) {
            fileSystem.close();
        }
        if (fileOutputStream != null) {
            fileOutputStream.close();
        }
    }
}

Kindly let me know for any other help.

Cheers.!!