Alternative/faster method to slow StreamWriter to print strings to a file in C#

Question

Some help from experts. I'm trying to use the function below to print strings to a file. When I use Console.Write() or Console.WriteLine() the output file grows up 3MB or 4MB per seconds, but when I try to use StreamWriter or File.AppendAllText the output in the way shown below, the file grows up only in 20KB or 30KB per second.

Why the print speed decreases too much when I use StreamWriter instead of Console.WriteLine()? What method should I use to write to a file maintaining the same speed of Console.WriteLine()?

    public static void PrintFunction()
    {            
        //using (StreamWriter writer = File.AppendText(@"C:\OuputFile.txt"))
        using (StreamWriter writer = new StreamWriter(@"C:\OuputFile.txt", true))
        {            
            //Console.Write("This is "); // Print speed is about 3MB-4MB per second
            writer.Write("This is "); //Print decreases to 20KB-30KB per second
            //File.AppendAllText(@"C:\OuputFile.txt", "This is "); Print decreases to 20KB-30KB per second

            // SOME CODE
            // SOME CODE

            //Console.WriteLine("the first line"); // Print speed is about 3MB-4MB per second
            writer.WriteLine("the first line"); // Print decreases to 20KB-30KB per second
            //File.AppendAllText(@"C:\OuputFile.txt", "the first line"); // Print decreases to 20KB-30KB per second
        }
    }

Update: When I say I'm using Console.WriteLine() I mean, I'm using Console.WriteLine() inside the code but to save those prints in a file, I'm redirecting the output like this:

 MyProgram.exe inputfile > outputfile.txt

I know the difference of memory and hard disk, but why when I use Console.WriteLine() redirecting the output as mentioned above (is printing to hard disk), the printing is more than 1000 times faster than using StreamWriter?

I've tried increasing the buffer size like below, but the speed of printing doesn't growp up.

using (StreamWriter writer = new StreamWriter(@"C:\OuputFile.txt", true, Encoding.UTF8, 65536))

Update 2:

Hello to all, Thanks for all the help, you were rigth!!!. Following all your suggestions and examples I defined StreamWriter outside the PrintFunction and this time the writer process is called only once and the output file remains open till the end and in this way the printing process speed is the same as Console.WrileLine().

I've passed the writer as argument of the function like below and it works. I've tested with a buffer size of 4KB, 64KB and with default values like shown below and the faster result was when I set explicitely used buffer of 4096 bytes. The function was called a little bit more than 10 million times and output file was 670 MB.

*StreamWriter(@"C:\OuputFile.txt", true, Encoding.UTF8, 4096)  --> 660845.1181 ms --> 11.0140853 min
StreamWriter(@"C:\OuputFile.txt", true, Encoding.UTF8, 65536) --> 675755.0119 ms --> 11.2625835 min
StreamWriter(@"C:\OuputFile.txt")                             --> 712830.3706 ms --> 11.8805061 min*

Thanks again for the help.

Regards

The code looks like this:

public static void ProcessFunction()
{
    StreamWriter writer = new StreamWriter(@"C:\OuputFile.txt", true, Encoding.UTF8, 4096);
    while ( condition)
    {
        PrintFunction(writer);          
    }
    if( writer != null )
    {
        writer.Dispose();
        writer.Close();
    }       
}

public static void PrintFunction(StreamWriter writer)
{            
    //SOME CODE                 
    writer.Write("Some string...");         
    //SOME CODE
}

Answer 1

I profiled this and it looks like it is completely the opposite. I was able to get about .25GB/s written to a standard 10K rpm drive (no SSD). It looks like you're calling this function a lot and writing to the file by connecting to it new each time. Try something like this (I snipped this together quickly from a piece of old console logging code, so it might be a bit buggy, and error handling is certainly not complete):

public static class LogWriter
{
    // we keep a static reference to the StreamWriter so the stream stays open
    // this could be closed when not needed, but each open() takes resources
    private static StreamWriter writer = null;
    private static string LogFilePath = null;

    public static void Init(string FilePath)
    {
        LogFilePath = FilePath;
    }

    public static void WriteLine(string LogText)
    {
        // create a writer if one does not exist
        if(writer==null)
        {
            writer = new StreamWriter(File.Open(LogFilePath,FileMode.OpenOrCreate,FileAccess.Write,FileShare.ReadWrite));
        }
        try
        {
            // do the actual work
            writer.WriteLine(LogText);
        }
        catch (Exception ex)
        {
            // very simplified exception logic... Might want to expand this
            if(writer!=null)
            {
                writer.Dispose();
            }
        }
    }

    // Make sure you call this before you end
    public static void Close()
    {
        if(writer!=null)
        {
            writer.Dispose();
            writer = null;
        }
    }
}

Answer 2

Why the print speed decreases too much when I use StreamWriter instead of Console.WriteLine()?

When you redirect the command output to a file, the write-only access of the output file was acquired by cmd.exe at once whatever how many times you call PrintFunction() with console.Write()

if you use stream writer in the PrintFunction(), the writer was initialized each time, try to access the file, write a line then release the file handle. The overhead kills the performance.

What method should I use to write to a file maintaining the same speed of Console.WriteLine()?

You can try one of the following

1. Buffer all your output in memory (e.g. using StringBuilder), then write to the file at once
2. Passing the StreamWriter object to the PrintFunction() to avoid the overhead. Proper handle the StreamWriter.Close() at the end