The right way to use BufferdOutputStream to write a large number of different small files

Question

I have a need to write a large number of different small files(about 30kb per file).Below is my java code:

 for(every file){
    File destFile = new File(fileName);
    try {
        FileOutputStream fos = new FileOutputStream(destFile);
        BufferedOutputStream bos = new BufferedOutputStream(fos);
        // System.out.println(dr.readLine());
        bos.write(result.getBytes(
                "UTF-8"));

    } catch (FileNotFoundException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (UnsupportedEncodingException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

Of course ,for every file,I have to new a File object.But is it necessary to new fileOutputStream and BufferdOutputStream object for every distinct file?Is there a more efficient way to write a large number of small files?

No, you can't share `FileOutputStream` between files. – Sotirios Delimanolis Dec 28 '13 at 02:12 — Sotirios Delimanolis, Dec 28 '13 at 02:12

score 1 · Answer 1 · edited Dec 28 '13 at 02:37

1

You can't share the FileOutputStream, as Sotirios says. The problem with your method is that you're creating a new File every iteration, without closing them. This may cause your program to overcome the limit of open files allowed.

On linux use:

cat /proc/sys/fs/file-max

to read the maximum number of files you can open.

and:

ulimit -n unlimited

to remove that limit (be careful and investigate other ulimit options).

edited Dec 28 '13 at 02:37

Martin Dinov

8,757
3
29
41

answered Dec 28 '13 at 02:22

elbuild

4,869
4
24
31

1

thx @Martin for the edit. Time to go to bed in Italy... 3.30 am :) – elbuild Dec 28 '13 at 02:38

score 1 · Answer 2 · edited Dec 28 '13 at 02:51

1

No. If you use FileOutputStream wrapped into BufferedOutputStream, you must create both instance per file.

But if you use java 7, you can use Files#write(Path path, byte\[\] bytes, OpenOption... options) without creating both instance. This class has introduced from java 7.

edited Dec 28 '13 at 02:51

PM 77-1

12,933
21
68
111

answered Dec 28 '13 at 02:25

t_ozawa

274
1
6

The right way to use BufferdOutputStream to write a large number of different small files

2 Answers2