Writing file to HDFS using Java

Question

I'm trying to write a file to HDFS, the file get created but it is empty on the cluster, however when I run the code locally it works like a charm.

here's my code :

FSDataOutputStream recOutputWriter = null;
FileSystem fs = null;
try {
    //OutputWriter = new FileWriter(outputFileName,true);
    Configuration configuration = new Configuration();
    fs = FileSystem.get(configuration);
    Path testOutFile = new Path(outputFileName);
    recOutputWriter = fs.create(testOutFile);

    //outputWriter = new FileWriter(outputFileName,true);
} catch (IOException e) {
    e.printStackTrace();
}
recOutputWriter.writeBytes("======================================\n");
recOutputWriter.writeBytes("OK\n");
recOutputWriter.writeBytes("======================================\n");

if (recOutputWriter != null) {
    recOutputWriter.close();
}
fs.close();

did I miss something ?

flush data ? https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/fs/FSDataOutputStream.html#hflush%28%29 — Pierre, Jun 29 '16 at 11:05
@Pierre thanks for your reply, even when I flush the data, the file still blank! — Jay, Jun 29 '16 at 11:10
What is `outputFileName`'s value? and did you try `fs` and `testOutFile` instantiation as this(with hdfs URI) http://stackoverflow.com/questions/16000840/write-a-file-in-hdfs-with-java — Ronak Patel, Jun 29 '16 at 13:31
@BigDataLearner , outputFileName can be a path like /files/out.txt OR out.txt — Jay, Jun 29 '16 at 14:52
@BigDataLearner I've tried the AddResources but didn't succeed — Jay, Jun 29 '16 at 16:29
can you try using `recOutputWriter.write()` instead of `recOutputWriter.writeBytes()` ? — Ronak Patel, Jun 29 '16 at 18:24
didn't try that.. should that make any difference ? recOutputWriter.write("OK".getBytes()); ? — Jay, Jun 29 '16 at 18:28
`recOutputWriter.write("OK")` - http://docs.oracle.com/javase/7/docs/api/java/io/FilterOutputStream.html?is-external=true#write(byte[]) - also see this example - http://mund-consulting.com/Blog/file-operations-in-hdfs-using-java/ - (P.S: sorry I don't have to my cluster so I cant test and give you exact answer) — Ronak Patel, Jun 29 '16 at 18:37
adding this solved the problem : System.setProperty("HADOOP_USER_NAME", "vagrant"); — Jay, Jun 30 '16 at 06:29

score 1 · Accepted Answer · answered Jun 30 '16 at 06:35

1

In order to write data to a file after creating it on the cluster I had to add :

System.setProperty("HADOOP_USER_NAME", "vagrant");

Refrence - Writing files to Hadoop HDFS using Scala

answered Jun 30 '16 at 06:35

Jay

717
11
37

Hi can I bother you with some noob questions? I didn't wan to open a new thread for something like this. But how do you actually write to HDFS? Is there some tutorial or blog I can follow? I absolutely beginner and have managed to create a cluster after 36 hours of banging my head at the screen. Thanks! – usamazf Dec 12 '17 at 05:15

Writing file to HDFS using Java

1 Answers1