Remove last x lines from a streamreader

Question

I need to read in all but the last x lines from a file to a streamreader in C#. What is the best way to do this?

Many Thanks!

Answer 1

If it's a large file, is it possible to just seek to the end of the file, and examine the bytes in reverse for the '\n' character? I am aware that \n and \r\n exists. I whipped up the following code and tested on a fairly trivial file. Can you try testing this on the files that you have? I know my solution looks long, but I think you'll find that it's faster than reading from the beginning and rewriting the whole file.

public static void Truncate(string file, int lines)
{
    using (FileStream fs = File.Open(file, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None))
    {
        fs.Position = fs.Length;

        // \n \r\n (both uses \n for lines)
        const int BUFFER_SIZE = 2048;

        // Start at the end until # lines have been encountered, record the position, then truncate the file
        long currentPosition = fs.Position;
        int linesProcessed = 0;

        byte[] buffer = new byte[BUFFER_SIZE];
        while (linesProcessed < linesToTruncate && currentPosition > 0)
        {
            int bytesRead = FillBuffer(buffer, fs);

            // We now have a buffer containing the later contents of the file
            for (int i = bytesRead - 1; i >= 0; i--)
            {
                 currentPosition--;
                 if (buffer[i] == '\n')
                 {
                     linesProcessed++;
                     if (linesProcessed == linesToTruncate)
                         break;
                 }
            }
        }

        // Truncate the file
        fs.SetLength(currentPosition);
    }
}

private static int FillBuffer(byte[] buffer, FileStream fs)
{
    if (fs.Position == 0)
        return 0;

    int bytesRead = 0;
    int currentByteOffset = 0;

    // Calculate how many bytes of the buffer can be filled (remember that we're going in reverse)
    long expectedBytesToRead = (fs.Position < buffer.Length) ? fs.Position : buffer.Length;
    fs.Position -= expectedBytesToRead;

    while (bytesRead < expectedBytesToRead)
    {
        bytesRead += fs.Read(buffer, currentByteOffset, buffer.Length - bytesRead);
        currentByteOffset += bytesRead;
    }

    // We have to reset the position again because we moved the reader forward;
    fs.Position -= bytesRead;
    return bytesRead;
}

Since you are only planning on deleting the end of the file, it seems wasteful to rewrite everything, especially if it's a large file and small N. Of course, one can make the argument that if someone wanted to eliminate all lines, then going from the beginning to the end is more efficient.

Answer 2

You don't really read INTO a StreamReader. In fact, for the pattern you're asking for you don't need the StreamReader at all. System.IO.File has the useful static method 'ReadLines' that you can leverage instead:

IEnumerable<string> allBut = File.ReadLines(path).Reverse().Skip(5).Reverse();

The previous flawed version, back in response to the comment thread

List<string> allLines = File.ReadLines(path).ToList();
IEnumerable<string> allBut = allLines.Take(allLines.Count - 5);

Answer 3

Since you are referring to lines in a file, I'm assuming it's a text file. If you just want to get the lines you can read them into an array of strings like so:

string[] lines = File.ReadAllLines(@"C:\test.txt");

Or if you really need to work with StreamReaders:

using (StreamReader reader = new StreamReader(@"C:\test.txt"))
        {
            while (!reader.EndOfStream)
            {
                Console.WriteLine(reader.ReadLine());
            }
        }