Best way to write huge string into a file

Question

In C#, I'm reading a moderate size of file (100 KB ~ 1 MB), modifying some parts of the content, and finally writing to a different file. All contents are text. Modification is done as string objects and string operations. My current approach is:

1. Read each line from the original file by using StreamReader.
2. Open a StringBuilder for the contents of the new file.
3. Modify the string object and call AppendLine of the StringBuilder (until the end of the file)
4. Open a new StreamWriter, and write the StringBuilder to the write stream.

However, I've found that StremWriter.Write truncates 32768 bytes (2^16), but the length of StringBuilder is greater than that. I could write a simple loop to guarantee entire string to a file. But, I'm wondering what would be the most efficient way in C# for doing this task?

To summarize, I'd like to modify only some parts of a text file and write to a different file. But, the text file size could be larger than 32768 bytes.

== Answer == I'm sorry to make confusin to you! It was just I didn't call flush. StremWriter.Write does not have a short (e.g., 2^16) limitation.

Answer 1

StreamWriter.Write

does not

truncate the string and has no limitation.

Internally it uses String.CopyTo which on the other hand uses unsafe code (using fixed) to copy chars so it is the most efficient.

Answer 2

The problem is most likely related to not closing the writer. See http://msdn.microsoft.com/en-us/library/system.io.streamwriter.flush.aspx.

But I would suggest not loading the whole file in memory if that can be avoided.

Answer 3

can you try this :

    void Test()
    {
        using (var inputFile = File.OpenText(@"c:\in.txt"))
        {
            using (var outputFile = File.CreateText(@"c:\out.txt"))
            {
                string current;
                while ((current = inputFile.ReadLine()) != null)
                {
                    outputFile.WriteLine(Process(current));
                }
            }
        }
    }

    string Process(string current)
    {
        return current.ToLower();
    }

It avoid to have to full file loaded in memory, by processing line by line and writing it directly

Answer 4

Well, that entirely depends on what you want to modify. If your modifications of one part of the text file are dependent on another part of the text file, you obviously need to have both of those parts in memory. If however, you only need to modify the text file on a line-by-line basis then use something like this :

using (StreamReader sr = new StreamReader(@"test.txt"))
{
    using (StreamWriter sw = new StreamWriter(@"modifiedtest.txt"))
    {
        while (!sr.EndOfStream)
        {
            string line = sr.ReadLine();
            //do some modifications
            sw.WriteLine(line);
            sw.Flush(); //force line to be written to disk
        }
    }
}

Answer 5

Instead of of running though the hole dokument i would use a regex to find what you are looking for Sample:

public List<string> GetAllProfiles()
    {
        List<string> profileNames = new List<string>();
        using (StreamReader reader = new StreamReader(_folderLocation + "profiles.pg"))
        {
            string profiles = reader.ReadToEnd();
            var regex = new Regex("\nname=([^\r]{0,})", RegexOptions.IgnoreCase);
            var regexMatchs = regex.Matches(profiles);
            profileNames.AddRange(from Match regexMatch in regexMatchs select regexMatch.Groups[1].Value);
        }
        return profileNames;
    }