Appending to a file without closing it in HDFS

Asked Mar 08 '18 at 18:32

Active Mar 08 '18 at 19:00

Viewed 297 times

I am trying to append some text in HDFS, but it turns out that I have to close the output stream fin_a.close() after every appending. Is there any way which I can append to a file without closing it?

try{
       Path filenamePath = new Path(appId + "_" + "file.txt");
       FSDataOutputStream fin_a = null;
       FSDataOutputStream fin_c = null;
       FileSystem fs = null;
       for(String st : list){
           fs = FileSystem.newInstance(config);
           if(fs.exists(filenamePath)){
             fin_a = fs.append(filenamePath);
             fin_a.writeUTF(st + "\n"));
             fin_a.close();
           } else {
                fin_c = fs.create(filenamePath);
                fin_c.writeUTF(st + "\n"));
                fin_c.close();
             }
       }
       fs.close();
   } catch (IOException e) {
           }

If that's not possible, does anyone have an idea how I can concatenate all the text in the list and write to the file once?

edited Mar 08 '18 at 19:00

Community

asked Mar 08 '18 at 18:32

Self

What about fin_a.hflush()? – alexq Mar 08 '18 at 19:03
The same thing. I only see one append. – Self Mar 08 '18 at 19:36
Try this: https://stackoverflow.com/questions/32231105/why-is-hsync-not-flushing-my-hdfs-file – alexq Mar 08 '18 at 19:49
1

Use a StringBuilder or a StringJoiner, then write out a single file at once... Also, no need to create the filesystem within the loop – OneCricketeer Mar 09 '18 at 01:20
Yes, this what I did. The for-loop is now faster. Please write your comment as an answer. – Self Mar 13 '18 at 02:15

Appending to a file without closing it in HDFS

0 Answers0