Advanced Reading of File

Question

I'm sure we're all familiar and probably use the plethora of code provided in books, online, etc. in reading a file using C#. Something as simple as...

StringBuilder fiContents = new StringBuilder();
using (StreamReader fi = new StreamReader(@"C:\a_file.txt"))
{
    while (!fi.EndOfStream)
    {
        fiContents.Append(fi.ReadLine); 
    }
}

Or maybe something as short as...

using (StreamReader fi = new StreamReader(@"C:\a_file.txt"))
    fiContents.Append(fi.ReadToEnd());

Now let's go Super Saiyan for a moment and do really fancy stuff like have a BackgroundWorker which will allow us show a loading image (this what I'll use), provide a process countdown timer or ProgressBar.

public void ReadFile(string filename)
{
    BackgroundWorker procFile = new BackgroundWorker();
    // Progress 1: If we want to show the progress we need to enable the following property
    // procFile.WorkerReportsProgress = true;

    profile.DoWork += new DoWorkEventHandler((object obj, DoWorkEventArgs ev) =>
    {
        StringBuilder fiContents = new StringBuilder();

        using (StreamReader fi = new StreamReader(filename))
        {
            while (!fi.EndOfStream)
            {
                // Progress 2: Report the progress, this will be dealt with by the respective handler (below).
                // procFile.ReportProgress((int)(fi.BaseStream.Length / fi.BaseStream.Position) / 100);

                fiContents.Append(fi.ReadLine);
            }
        }

        ev.Result = fiContents;
    }

    /* Progress 3: The handler below will take care of updating the progress of the file as it's processed. 
    procFile.ProgressChanged += new ProgressChangedEventHandler((object obj, ProgressChangedEventArgs ev) =>
    {
        // Progress 4: Do something with the value, such as update a ProgressBar. 
        // ....
    }
    */

    procFile.RunWorkerCompleted += new RunWorkerCompletedEventHandler((object obj, RunWorkerCompletedEventArgs ev) =>
    {
         // Do something with the result (ev.Result), bearing in mind, it is a StringBuilder and the ev.Result is an object. 
         StringBuilder result = ev.Result as StringBuilder; 

         // ....
    }
}

+++++ +++++ +++++ +++++

Time for the actual question... The above was a warm-up and to show a current level of understanding so I don't face these as prospective answers.

I'm pretty much doing the last code example given above (i.e. using a BackgroundWorker) and dumping the contents of what is read to a RichTextBox. Simple stuff really.

The problem I'm facing however is processing large files (e.g. ~222MB). The case being just taking a .txt, reading it, pushing the result of it built through a StringBuilder into the RichTextBox. It cannot load the file, I get an OutOfMemoryException. One way around this, which takes a considerable amount (and still doesn't load the file) is iterating through the string and adding each character (as a char) from the file StringBuilder.

I've always used the most basic and straightforward means of reading files (such as the examples given above), but does anyone have any guidance on how to improve on this? Ways of processing extremely large files? etc.

Even as a discussion piece, I'd welcome your ideas.

+++++ +++++ +++++ +++++

Edit 1 (@TaW): the exception was thrown when trying to put the string into the RichTextBox...

FileProcessing.RunWorkerCompleted += new RunWorkerCompletedEventArgs((object obj, RunWorkerCompletedEventArgs e) =>
{
    // 'Code' is the RichTextBox in question...

    Code.Text = "";

    if (e.Result is StringBuilder)
    {
        Code.Text = (e.Result as StringBuilder).ToString();
    }
}

Answer 1

Is there a restriction you have that requires you to use a RichTextBox as the control to display your content? This control is not virtualized and will cause you performance (and by the looks of it memory error) issues.

There are a family of document viewing controls that are better designed for displaying large documents. Various controls exists depending on your needs (fixed, flowing via page or scrolling). In addition, you get searching, printing, zooming and a few other features that are often useful for viewing large documents.

Answer 2

Have you tried the MemoryMapped ,
Its pretty useful lib for handling large files

Answer 3

this is not about advanced reading but about hitting the capacity limits of (Winforms) controls. Maybe you can get it to work in WPF, but in Winforms neither a RichTextBox nor a TextBox can hold such a large amount of lines/text.

I advise you to redesign this to present the data to the users in smaller chunks. It is not that they would want to scroll through 100.000+ lines. Processing them in memory is not an issue; here 200MB is not large at all; you can for example easily search in it in memory etc..