FileStream.Close() is not closing the file handle instantly

Question

I am reading source file in chunks and pass it to WCf service to write down on some remote SMB. I am keeping open the FileStream until all data is written.

Opening and closing file handle multiple time decrease performance so I am following this approach.

Once all data is written, I call CloseHandle(). Then I may need to perform some other operation on same file by calling DoSomeOperation(). As I have closed the file handle in CloseHandle() function, but I get the error "file is in use with some other process" in DoSomeOperation(). If I call DoSomeOperation() after some delay then issue is not there.

Please help us to close the file handle instantly as I call FileStream.Close().

This code snippet is part of a big program, so I can't mention all the code here.

//In WCF service
FileStream fs = null;
public void AppendBytes(string fileName, byte[] data, long position)
{
    try
    {
        if (fs==null)//In first call, open the file handle
            fs = System.IO.File.Open(fileName, System.IO.FileMode.Append, System.IO.FileAccess.Write, System.IO.FileShare.None);

        fs.Write(data, 0, data.Length);
    }
    catch (Exception ex)
    {
        //Close handle in case of error
        if (fs != null)
            fs.Close();
    }
}

public void CloseHandle()
{
    //Close handle explicitly
    if (fs != null)
        fs.Close();
}

public void DoSomeOperation(string fileName)
{
    using (FileStream fsO = System.IO.File.Open(fileName, System.IO.FileMode.Append, System.IO.FileAccess.Write, System.IO.FileShare.None))
    {
        //Do something with file here, this is atomic operation so I am opening FileStream with 'using' to dispose at operation is over
    }
}

//In client
public void CallerFunction()
{
    //Read Data from sourceFile in chunk and copy to target file using WCF.AppendBytes on another machine
    WCF.AppendBytes(filename, data, pos);
    WCF.CloseHandle();
    WCF.DoSomeOperation(filename); //I get error here that file is in use with some other process. if I put a thread.sleep(1000) just before this statement then all works fine.
}

I have written a small test code to reproduce the same scenario on console application: Just Call TestHandleClose() from Main(), it will give error after some cycles.

 static  void TestHandleClose()
        {
            int i = 0;
            try
            {

                if (File.Exists(@"d:\destination\file2.exe"))
                    File.Delete(@"d:\destination\file2.exe");

                byte[] data = null;
                int blocksize = 10 * 1024 * 1024;

                for( i=0;i<100;i++)
                {
                    using (FileStream fr = File.Open(@"d:\destination\File1.zip", FileMode.Open, FileAccess.Read, FileShare.None))
                    {
                        data = new byte[blocksize];
                        fr.Read(data, 0, blocksize); //We are reading the file single time but appending same data to target file multiple time.

                        using (FileStream f = File.Open(@"d:\destination\file2.exe", FileMode.Append, FileAccess.Write, FileShare.None))
                        {
                            f.Write(data, 0, data.Length); //We are writing same data multiple times.
                            f.Flush();
                            f.Close();
                        }
                    }

                }

                if (File.Exists(@"d:\destination\file2.exe"))
                    File.Delete(@"d:\destination\file2.exe");

            }
            catch (Exception ex)
            {
                throw;
            }
        }

Try and add in fs.Flush() before all fs.Close() methods. Let us know if that works. — Praval 'Shaun' Tirubeni, May 28 '15 at 09:49
I have already tried that but no impact. Close internally calls Flush(). — Romil Kumar Jain, May 28 '15 at 09:50
By any chance `CallerFunction` is called in multiple threads? — Sriram Sakthivel, May 28 '15 at 09:52
If you have an antivirus, can you try to disable it for testing purpose? — Matteo Umili, May 28 '15 at 09:52
If I put thread,sleep(1000) just before calling DoOtherOperation() then no issues — Romil Kumar Jain, May 28 '15 at 09:54
My application has digital certification and I have only MS Essential AV enabled. — Romil Kumar Jain, May 28 '15 at 09:56
I'm immediately suspicious of code like this where `fs` is a shared variable. E.g. if there are two simultaneous calls into `AppendBytes` on the same instance, they might both observe `fs` to be null, both open a new `Stream`, both compete to set `fs` and you leak one stream until its garbage collected. Whether that's possible here is tricky to say, and depends on your instancing/threading options. — Damien_The_Unbeliever, May 28 '15 at 09:58
@Damien_The_Unbeliever, No threading is used, so only single call is made at a time. if you check the code at end of my post, it also gives error after 4-5 cycles. — Romil Kumar Jain, May 28 '15 at 10:00
Instead of showing us bits and pieces (especially where one code snippet apparently contains pieces from two completely separate programs), please try to create a [MCVE](http://stackoverflow.com/help/mcve). — Damien_The_Unbeliever, May 28 '15 at 10:00
I tested the same code where AV is not installed. same results. — Romil Kumar Jain, May 28 '15 at 10:06
I don't think you understand how `FileShare` works - you're explicitly allowing write-only streams to work on the same file. You really want to use `FileShare.None`. — Luaan, May 28 '15 at 10:06
@Luaan While Appending bytes, I want no one should use that file. so I am using FileShare.None — Romil Kumar Jain, May 28 '15 at 10:08
Well, the thing about WCF is that for every client call a new service instance is created usually. Are you sure you configured WCF to use a static instance? In any other case `CloseHandle` will do nothing as `fs` is null and the file will remain open from the first service instance as it is never closed. — Thorsten Dittmar, May 28 '15 at 10:08
Your sample code runs fine with LinqPad. Accessing a 4 MB big xml file. — Jehof, May 28 '15 at 10:09
Unable to reproduce on Windows 7, running the tight while-loop at the bottom does not crash, have had it running for 5 minutes straight now. You probably need to use procmon or similar to watch what really happens. — Lasse V. Karlsen, May 28 '15 at 10:09
Your sample code uses `FileShare.Write`, not `FileShare.None`. So either fix your sample code, or... fix your real code :) — Luaan, May 28 '15 at 10:11
@Romil So which operation starts the session, which one stops it? — Thorsten Dittmar, May 28 '15 at 10:13
@ThorstenDittmar, AppendBytes is called many times depending upon source file size and chunk size. Then Close Handle then DoOtherOperation() — Romil Kumar Jain, May 28 '15 at 10:16
Okay, I can confirm Lasse's results - tried with both local files and remote files, and it just works. Either you're using some non-compliant SMB remote, or someone else is using that file. — Luaan, May 28 '15 at 10:16
IIRC when you use PerSession mode, you need one operation that starts the session and one that stops it - maybe that's part of the problem, too? — Thorsten Dittmar, May 28 '15 at 10:56
@ThorstenDittmar I have added some test code, so you can repro the issue on your machine also. I am just trying to open the target files for appending data. — Romil Kumar Jain, May 28 '15 at 12:22
@Luaan I have added some test code, so you can repro the issue on your machine also. I am just trying to open the target files for appending data. — Romil Kumar Jain, May 28 '15 at 12:22

score 5 · Accepted Answer · answered May 28 '15 at 12:39

The final sample code helped, I'm pretty sure I have your problem.

The thing is, even the latest sample code fails to reproduce. However, it shows another thing that you probably missed - the file you're writing is .exe. Why is this a problem? Well, there's a couple of reasons, but one of those is that when you list the directory where the .exe file is using explorer, explorer goes ahead and tries to read it (to get the icon). In this short time, the file cannot be opened with FileShare.None (and in fact, FileShare.Read probably will not help either, since it's quite likely whoever opened it didn't specify FileShare.ReadWrite).

So yet again, FileStream is closed just fine, and works well (get rid of the Flush and Close calls, though - they're wasting performance, and useless). The problem is that another process tries to read the file in the meantime. It might be some file manager like in my case (Explorer, Total Commander, ...), it might be some FileSystemWatcher you have somewhere, it might be a virus scanner (though most virus scanners nowadays use shadow copies), it might be something that automatically creates thumbnails for images, etc.. But the code you posted yourself simply doesn't cause the problem - it's someone else grabbing your file.

There's basically two options you have - either keep the file opened the whole time you need it, or treat the IOException as temporary and retry a few times in a given interval. That's what you should be doing anyway, instead of relying on the path being happy - most readers only allow concurrent reads, not writes.

you are amazing, it worked like miracle. The issue is only with extension .exe. I put Flush(), Close() in using block to satisfy other developers. They keep suggesting these ideas :). so incase of IO exception, I will try to access the same file handle for max 1 sec in loop with some Thread.Sleep(50) before declaring that file is being used by some other process. — Romil Kumar Jain, May 28 '15 at 12:58
@Romil Yeah, `Flush` is pretty much guaranteed to make your performance worse - the `Dispose` called automatically as `using` comes out of scope will take care of flushing and closing just fine. When you `Flush`, you basically say "throw away all those buffers that make everything work fast" (and it doesn't even ensure that the data is flushed anyway, since it's actually asynchronous, not blocking). It doesn't make a difference when you `Flush` just once before closing the file, but it's unnecessary and possibly confusing :) — Luaan, May 28 '15 at 13:05

score 0 · Answer 2 · answered Feb 02 '17 at 22:19

In my case it was the FileShare mode that was missing. I used to read/write from multiples sources in the same file in a short period of time . Sometimes it passed good and sometimes it blocked (another process was holding the file) even though the filestream was released and i was using a lock and a using(). Everything solved when i added the FileShare mode ReadWrite.

using (FileStream fileStream = new FileStream(file, mode, access, FileShare.ReadWrite))
{...

Please read the answer from Luaan. It was a totally diffrent scenario. — Romil Kumar Jain, Feb 03 '17 at 06:59

FileStream.Close() is not closing the file handle instantly

2 Answers2

Linked