Open file descriptors in lsof as (deleted)

Question

I have in my java process code that opens a file for writing using Files.newOutputStream which is inside the try-with-resources statement. After I finishing writing the files (all written in the same way) and closed. They deleted together with the folder that files are inside of it.

String folder = "folder"
    try(OutputStream out = Files.newOutputStream(Paths.get(folder + '/' + fileName), StandardOpenOption.CREATE, StandardOpenOption.WRITE)) {
        //...
        out.write(buffer);
        //...
    } catch(IOException excp) {
        excp.printStackTrace();
    }
    FileUtils.forceDelete(new File(folder));

But, when I'm looking in the lsof (I'm running CentOS) I see that those files (and there could be 20k or 30k) are shown as (deleted) in lsof and means that they are open. I can't understand how closed file can remains open and eventually I get too many open file descriptors. So, I know that I can raise the limit, but I want to properly close or to make file closed instead of increasing limits.

Thanks !!!

Please edit your question to include the relevant source code from your java program. Ie, the code that creates these files, writes to them, and deletes them. — Kenster, May 10 '16 at 22:26
@lincolnadym FileUtils.forceDelete is Apache FileUtils, correct. — SlavaG, May 11 '16 at 05:40
@SlavaG, based on the comments below, I think the reason you are getting all those lingering File handles in lsof is that new File(folder). — lincolnadym, May 11 '16 at 20:24
@SlavaG, That line of code is your issue, FileUtils.forceDelete(new File(folder)); I downloaded the apache-commons-io source and checked the FileUtils.forceDelete() code...It creates an array of File[] objects, which is creating all those File Handles in lsof. — lincolnadym, May 11 '16 at 20:40
@SlavaG, perhaps I'm mis-reading the code in FileUtils. But forceDelete() ends up calling cleanDirectory(), which in turn creates an array of File[] objects. It then calls forceDelete() for each File[] object. I guess I don't know what you mean by "file descriptor"? — lincolnadym, May 11 '16 at 20:50
I guess it really doesn't matter about "file descriptors". In the end, in order for Java to actually "delete" the file, it HAS to create something in the OS layer that has write functionality to the file, which is where your File handle in lsof is probably coming from. — lincolnadym, May 11 '16 at 20:57
@lincolnadym Not exactly , File in java just an object that don't open any file handler in OS , it has inside only abstraction layer for the file system and delete operation is done without opening file handler (descriptor). — SlavaG, May 11 '16 at 21:00
There's nothing obviously wrong with the code that you posted. The simplest explanation is that you're problem is somewhere else. Some other process, or some other part of this program, is opening the files and failing to close them. — Kenster, May 30 '16 at 13:50
Hi SlavaG, did you ever figure this out? I'm having the exact same issue haha. `df` shows empty space but `du` shows its full, because `lsof` list a ton of files that are marked `(deleted)` but still seem to be taking space. It eventually leads to an exception about no disk space that I can't figure out how to fix. — rococo, Mar 22 '19 at 07:31
Well, I found that files were deleted but other part of the code still holding them. So, closed it and then deleted solved the issue. But, also I switched from ext4 to XFS where no inodes issue where I have huge amount of small files, then you can find yourself with no space error, but actually with lot of free space, but no available inodes. — SlavaG, Mar 30 '19 at 11:05

lincolnadym · Answer 1 · 2016-05-11T01:20:53.473

-3

I think you aren't actually closing the files...for lsof it's a File Handle that's not getting closed.

Maybe add the finally {} block to your code :

OutputStream out = null
try {
    out = Files.newOutputStream(Paths.get(folder + '/' + fileName), StandardOpenOption.CREATE, StandardOpenOption.WRITE)) {
    //...
    out.write(buffer);
    //...
} catch(IOException excp) {
    excp.printStackTrace();
}
finally { 
    if ( out != null ) { out.close(); } 
} 
FileUtils.forceDelete(new File(folder));

Make sure you test the out for not null, otherwise it'll throw a NullPointerException when you try and close.

EDIT : Forgot to move the OutputStream outside the try{} catch{} finally {}...

edited May 11 '16 at 01:20

answered May 11 '16 at 01:11

lincolnadym

909
1
12
29

The way the try {Outputstream} pattern works, is it should automatically close the File once it gets out of scope. I wonder if the FileUtils call is actually creating a new File Handle on the File. – lordoku May 11 '16 at 01:24
Hmmm, what's new File (folder) create inside the FileUtils.forceDelete(new File(folder))? The code would 'suggest' a 'single' new File handle, but if folder is filled with say 100 files, does the FileUtils, create 100 new File() handles? – lincolnadym May 11 '16 at 01:32
@lincolnadym There is no such thing as a '`File` handle'. They are just objects, with no connection to the file system. – user207421 May 11 '16 at 01:58
@lincolnadym FileUtils.forceDelete is Apache FileUtils https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/FileUtils.html#forceDelete(java.io.File) – SlavaG May 11 '16 at 05:22
@EJP in Java, yes they are objects, but each File that is open creates an handle on the OS. – lordoku May 11 '16 at 16:10
@EJP, Yes, i know Java has no File handle. I should have clarified, I was speaking to the lsof command in the question. – lincolnadym May 11 '16 at 20:27
@lordoku Of course, but that has nothing to do with the `File` class. You don't even have to use it to open a file. – user207421 May 22 '16 at 01:55
@EJP, yes of course you don't have to open it. I was just trying to point out that not keeping track of the File object could result in File handle leaks in the OS. – lordoku May 23 '16 at 15:11
This isn't an improvement on the OP's original code. You should read up on how [try with resources](https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html) works. – Kenster May 30 '16 at 13:58

Open file descriptors in lsof as (deleted)

1 Answers1