Writing to a file in the background

Question

Given a method that writes to a text file

public void WriteToFile( ) {
    var file = "C:\\AsyncTest.txt";
    var writer = File.Exists( file ) ? File.AppendText( file ) : File.CreateText( file );
    writer.WriteLine( "A simulated entry" );
    writer.Close();
}

I need to simulate a scenario in which this method could be called in a loop, possibly dozens of times and must run asynchronously.

So I tried calling the method in a new thread like so (where writer is the class where WriteToFile lives)

//in a loop...
  Thread thread = new Thread( writer.WriteToFile );
  thread.Start(  );

Which works perfectly once, but throws an IO Exception that the file is being used by another process on subsequent iterations. Which makes perfect sense, actually, but I don't know how to work around it.

I tried using Join() like this

Thread thread = new Thread( writer.WriteToFile );
thread.Start(  );
thread.Join();

But that locks the calling thread until all the joined threads complete, which sort of defeats the purpose, no?

I tried using ThreadPool.QueueUserWorkItem(writer.WriteToFile);, but get the same IO exception.

I tried using a lock

private object locker = new object();

public void WriteToFile( ) {
  lock(locker){
    //same code as above
  }
}

But that had no apparent effect

I also tried using the Task class to no avail.

So how can I "stack up" these background threads to write to a single file without conflict, while not locking up the calling thread?

Answer 1

Another option is to create a queue. Have the main thread put strings on the queue and have a persistent background thread read the queue and write to the file. It's really easy to do.

private BlockingCollection<string> OutputQueue = new BlockingCollection<string>();

void SomeMethod()
{
    var outputTask = Task.Factory.StartNew(() => WriteOutput(outputFilename),
        TaskCreationOptions.LongRunning);

    OutputQueue.Add("A simulated entry");
    OutputQueue.Add("more stuff");

    // when the program is done,
    // set the queue as complete so the task can exit
    OutputQueue.CompleteAdding();

    // and wait for the task to finish
    outputTask.Wait();
}

void WriteOutput(string fname)
{
    using (var strm = File.AppendText(filename))
    {
        foreach (var s in OutputQueue.GetConsumingEnumerable())
        {
            strm.WriteLine(s);
            // if you want to make sure it's written to disk immediately,
            // call Flush. This will slow performance, however.
            strm.Flush();
        }
    }
}

The background thread does a non-busy wait on the output queue, so it's not using CPU resources except when it's actually outputting the data. And because other threads just have to put something on the queue, there's essentially no waiting.

See my blog, Simple Multithreading, Part 2 for a little bit more information.

Answer 2

You could use something like:

// To enqueue the write
ThreadPool.QueueUserWorkItem(WriteToFile, "A simulated entry");

// the lock
private static object writeLock = new object();

public static void WriteToFile( object msg ) {
    lock (writeLock) {
        var file = "C:\\AsyncTest.txt";

        // using (var writer = File.Exists( file ) ? File.AppendText( file ) : File.CreateText( file )) {
        // As written http://msdn.microsoft.com/it-it/library/system.io.file.appendtext(v=vs.80).aspx , File.AppendText will create the
        // file if it doesn't exist

        using (var writer = File.AppendText( file )) {
            writer.WriteLine( (string)msg );
        }
    }
}

And please, use using with files!

Answer 3

You could handle the locking like you tried, but you need to include the using statement:

private readonly object _lockObj = new Object();

public void WriteToFile( )
{
    var file = "C:\\AsyncTest.txt";
    lock (_lockObj)
    {
        using (StreamWriter writer = File.AppendText( file ))
        {
            writer.WriteLine( "A simulated entry" );
        }
    }
}

Further, you don't need to leverage CreateText because AppendText will create the file if it doesn't exist. Finally, I've had problems with this code before as well in that the lock will be released before Windows releases the resource. It's rare, but it happens, so I just add a little retry logic looking for that specific exception.

Answer 4

1. Dispose your streams.
2. Use TPL or async/await capabilities.

For example:

Task.Run(() => File.WriteAllText("c:\\temp\test.txt", "content"));

This will run a write operation asynchronously without you taking care of threads.

Also, streams and stream writers provide WriteAsync methods that you can use and await for.

UPDATE

To avoid "locking" problem simply don't do locking :) Locking happens if you try to write to the same file from different threads. You may use File.Open() methods and specify the mode so it will block a thread and wait until file is writable.

But blocking is bad. So I advice you, in case if you want to write from multiple threads, to create a queue and put your writing tasks into this queue. You can put safely from multiple thread (use ConcurrentQueue<T>). Then you consume this queue in a background task and just write to your file what you have in the queue - one item by one.

That's it: multiple publishers, one [file writing] consumer, super easy, no locks required.