C# - Most effective way to periodically read last part of a file

Question

I want to periodically read a log file that is also being written to. The program will be periodically reading the log file contents and parsing it to extract some values. But I do not want to read the whole file each time.

Is there a way to read a file from a particular line onwards?

For example at the first read the file has 100 lines. I note this value and next time I read I start reading from line 100 onwards and store the line number of the current file.

Is there an efficient way to do this? The log file will grow to about 100MB and I need to read about every 5 seconds. So it wouldn't be that efficient to read the complete file each time.

Any suggestion is greatly appreciated.

You might look at Tail.NET http://www.codeproject.com/Articles/7568/Tail-NET — itsmatt, May 01 '13 at 13:43
You could partition the file into multiple files or a DB intended for such a purpose. Although, I'm not sure how much benefit that provides with only 100MB file. Streams tend to have start indexes. Why not just use that? — P.Brian.Mackey, May 01 '13 at 13:46

score 6 · Accepted Answer · answered May 01 '13 at 13:52

6

I think you're looking for this, where offset is going to be how much you want backtrack. Reference: MSDN

using (FileStream fs = new FileStream(filepath, FileMode.Open, FileAccess.Read))
{
    fs.Seek(offset, SeekOrigin.End);
}

Now the filestream is pointing to however far back in the file you set 'offset' to be, and you can read from there.

answered May 01 '13 at 13:52

bland

1,968
1
15
22

...hah, I'm two minutes too slow ;) – J... May 01 '13 at 13:58
@bland Thank you. I will have look in to this method. Looks like this could work for me. – madu May 01 '13 at 14:09

score 3 · Answer 2 · answered May 01 '13 at 13:44

3

If the log is only appended, you can try opening a file in read-only mode without a lock. That way other processes can write to it while you read it.

var fs = new FileStream(path,FileMode.Open,FileAccess.Read, FileShare.ReadWrite);

answered May 01 '13 at 13:44

alex

12,464
3
46
67

If you remember the length of the file, you may be able to `Seek` to that location on the next read. – JosephHirn May 01 '13 at 13:53

score 3 · Answer 3 · answered May 01 '13 at 15:06

Seek can do this well. But I want to provide other onward way.

    public static void Read()
    {
        var fs = new FileStream(@"G:\test.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
        int lastReadCount = 0;
        while (true)
        {
            var totalCountOfFile = fs.Length;
            if (lastReadCount < totalCountOfFile)
            {
                var buffer = new byte[1024];
                int count = fs.Read(buffer, 0, buffer.Length);
                lastReadCount += count;
                Display(buffer);
            }
            Thread.Sleep(5000);
        }
    }

    private static void Display(byte[] buffer)
    {
        var text = Encoding.UTF8.GetString(buffer.Where(p=>p != 0).ToArray());
        Console.Write(text);
    }

J... · Answer 4 · 2013-05-17T03:02:46.207

For something fast and dirty, I use this. In this case it's a log dump - I don't really care about how many lines I get, I just want a bunch at the end (numBytes) :

cmdLogReader = new System.IO.StreamReader(cmdLogFileIn);

if (cmdLogReader.BaseStream.Length < (numBytes - 1)) {
    return cmdLogReader.ReadToEnd;
} else {
    cmdLogReader.BaseStream.Seek(-numBytes, System.IO.SeekOrigin.End);
    cmdLogReader.ReadLine();
    return cmdLogReader.ReadToEnd;         
}

You could always save the BaseStream.Length at the beginning and use that to calculate how far back to go next time (ie: numBytes becomes BaseStream.Length - previousBaseStreamLength or whatever), that would let sequential calls grab whatever has been added since the last read.

You may have to skip the ReadLine call if you do this since it's really only there to move up to the nearest line after backtracking a random amount. If you know you're going to land on a line boundary then you can just ReadToEnd.

This is a bit of a gritty implementation, but it's very fast which is why I use it.

Thank you. I wasn't aware of BaseStream.Length property and using this method to read the file. I will have a look in to this method. — madu, May 01 '13 at 14:10

C# - Most effective way to periodically read last part of a file

4 Answers4

Linked