Do I need to manually flush StreamWriter?

Question

I have the following scenario.

I am doing splitting functionality by reading huge csv file line by line. Each line have categoryId. Based on that Id I need to write this line into separate file.

To do this I am doing the following:

1. Reading the huge file line by line.
2. After reading each line I open a new stream based on the categoryId (only if the stream is not already opened). Write the line into the stream and then keep the stream open, because there might be more lines in in the huge file.
3. At the end after all lines from the huge file are processed I am closing all open streams. This forces flush and closes the connections.

My question is. Do I need to manually invoke Flush() on lets say -> every 100 lines recorded or this is something handled by StreamWriter itself. I read on the web that there is a buffer that automatically flushes when it is full, but I am not sure if this is true. My concern is that if it doesn't flush and waits for the end of the big file, I might end up with the whole file loaded in memory.

Here is part of the code to see what am talking about:

try
        {
            while (!reader.EndOfStream)
            {
                var line = await reader.ReadLineAsync();
                var locationId = line.Split(',')[0];
                var gdProjectId = GetGDProjectId(locationId);

                var blobName = $"{gdProjectId}/{DateTime.UtcNow.ToString("dd-MM-yyyy")}/{DateTime.UtcNow.ToString("HH-mm-ss")}-{Guid.NewGuid()}.csv";

                if (!openWriters.ContainsKey(gdProjectId))
                {
                    var blockBlobClient = containerClient.GetBlockBlobClient(blobName);
                    var newWriteStream = await blockBlobClient.OpenWriteAsync(true);
                    openWriters.Add(gdProjectId, new StreamWriter(newWriteStream, Encoding.UTF8));
                }

                var writer = openWriters[gdProjectId];
                await writer.WriteLineAsync(line);

                // SHOULD I MANUALLY INVOKE FLUSH ON EVERY {X} lines processed ?
                // TODO: Check if we need to manually flush or the streamwriter does it for us when the buffer is full.
                // await writer.FlushAsync();
            }
        }
        catch (Exception ex)
        {

            throw;
        }
        finally
        {
            // we are always closing the writers no matter if the operation is successful or not.
            foreach (var oStream in openWriters)
            {
                oStream.Value.Close();
            }
        }

Answer 1

Flush (in StreamWriter implementation) just sends data from buffer to underlying stream then calls Flush on underlying stream, i.e. (pseudo code):

underlyingStream.Write(GetDataFromBuffer());
bufferPosition = 0; // "clears" buffer
underlyingStream.Flush();

Buffer size is constant. By default it is something around 2-4KiB. But it can be set manually in constructor for larger values. Flush does not change buffer size. So calling Flush every 100 lines gives you nothing.

Q: "Do I need to manually invoke Flush() on lets say -> every 100 lines..."

No. It will not save you any memory. It will just write data to underlying stream earlier - i.e. it will not wait for buffer to be full.
Hint: if property AutoFlush is set to true, Flush will be automatically called after each WriteXYZ method call.

Q: "My concern is that if it doesn't flush and waits for the end of the big file, I might end up with the whole file loaded in memory."

Buffer size is constant. Calling Flush won't help.

But...

everything is true just from StreamWriter perspective.
Because StreamWriter just holds reference to some Stream instance, you cannot predict memory usage without knowing concrete implementation of that Stream instance (Stream is abstract).

You should create new question like: "Do I need to manually flush XyzStream"? (if there is not question like that already posted).