Read a Log file and pull out lines with specific text

Question

I have a log that is constantly being added to (like 1-5 lines a second or more) and the files can get rather large 20MB+

Example Log File:

[Wed Aug 26 00:01:48 2015] You try to kick a snake, but miss!
[Wed Aug 26 00:01:50 2015] A snake hits YOU for 3 points of damage.
[Wed Aug 26 00:01:51 2015] You punch a snake for 6 points of damage.
[Wed Aug 26 00:01:51 2015] A decaying skeleton tries to hit Clumsy, but misses!
[Wed Aug 26 00:01:53 2015] A snake hits YOU for 1 point of damage.
[Wed Aug 26 00:01:53 2015] A decaying skeleton tries to hit Clumsy, but misses!
[Wed Aug 26 00:01:54 2015] You gain experience!!
[Wed Aug 26 00:01:54 2015] You punch a snake for 6 points of damage.
[Wed Aug 26 00:01:54 2015] You have slain a snake!
[Wed Aug 26 00:01:56 2015] --You have looted a Snake Egg.--
[Wed Aug 26 00:01:56 2015] A decaying skeleton tries to hit Clumsy, but misses!
[Wed Aug 26 00:01:57 2015] --You have decided to not loot 2 item(s): Snake Fang. The item(s) will be available to anyone after the corpse(s) unlock.--
[Wed Aug 26 00:01:57 2015] --You have decided to not loot 2 item(s): Snake Scales. The item(s) will be available to anyone after the corpse(s) unlock.--
[Wed Aug 26 00:01:59 2015] a decaying skeleton's corpse splinters into hundreds of tiny bone fragments.
[Wed Aug 26 00:01:59 2015] Clumsy crushes a decaying skeleton for 7 points of damage.
[Wed Aug 26 00:01:59 2015] a decaying skeleton has been slain by Clumsy!
[Wed Aug 26 00:01:59 2015] Auto attack is on.
[Wed Aug 26 00:02:00 2015] Grug tells General:2, 'LFM AM/LORD'
[Wed Aug 26 00:02:00 2015] You punch a large rat for 5 points of damage.
[Wed Aug 26 00:02:00 2015] A large rat bites YOU for 1 point of damage.
[Wed Aug 26 00:02:00 2015] You kick a large rat for 1 point of damage.
[Wed Aug 26 00:02:02 2015] You punch a large rat for 2 points of damage.
[Wed Aug 26 00:02:03 2015] A large rat tries to bite YOU, but misses!
[Wed Aug 26 00:02:06 2015] You punch a large rat for 6 points of damage.
[Wed Aug 26 00:02:06 2015] A large rat bites YOU for 1 point of damage.
[Wed Aug 26 00:02:08 2015] a decaying skeleton clatters as it turns towards you.
[Wed Aug 26 00:02:08 2015] <SYSTEMWIDE_MESSAGE>: Gorenaire has been defeated by a group of hardy adventurers! Please join us in congratulating Kyrax along with everyone else who participated in this achievement!
[Wed Aug 26 00:02:08 2015] Clumsy tries to crush a decaying skeleton, but misses!
[Wed Aug 26 00:02:08 2015] A decaying skeleton tries to hit Clumsy, but misses!
[Wed Aug 26 00:02:09 2015] A large rat tries to bite YOU, but YOU dodge!
[Wed Aug 26 00:02:09 2015] You try to punch a large rat, but miss!
[Wed Aug 26 00:02:11 2015] Clumsy crushes a decaying skeleton for 8 points of damage.
[Wed Aug 26 00:02:11 2015] A decaying skeleton hits Clumsy for 2 points of damage.
[Wed Aug 26 00:02:11 2015] You kick a large rat for 1 point of damage.
[Wed Aug 26 00:02:12 2015] A large rat bites YOU for 1 point of damage.
[Wed Aug 26 00:02:13 2015] You gain experience!!
[Wed Aug 26 00:02:13 2015] You punch a large rat for 6 points of damage.
[Wed Aug 26 00:02:13 2015] You have slain a large rat!
[Wed Aug 26 00:02:13 2015] Clumsy crushes a decaying skeleton for 8 points of damage.
[Wed Aug 26 00:02:14 2015] A decaying skeleton hits Clumsy for 3 points of damage.
[Wed Aug 26 00:02:15 2015] --You have decided to not loot 1 item(s): Piece of Rat Fur. The item(s) will be available to anyone after the corpse(s) unlock.--
[Wed Aug 26 00:02:15 2015] --You have decided to not loot 2 item(s): Rat Whiskers. The item(s) will be available to anyone after the corpse(s) unlock.--
[Wed Aug 26 00:02:16 2015] a decaying skeleton's corpse splinters into hundreds of tiny bone fragments.
[Wed Aug 26 00:02:18 2015] Auto attack is on.
[Wed Aug 26 00:02:18 2015] a snake hisses and strikes!
[Wed Aug 26 00:02:18 2015] You punch a snake for 6 points of damage.
[Wed Aug 26 00:02:18 2015] A snake tries to hit YOU, but misses!
[Wed Aug 26 00:02:20 2015] You kick a snake for 1 point of damage.
[Wed Aug 26 00:02:21 2015] You try to punch a snake, but miss!
[Wed Aug 26 00:02:21 2015] A snake hits YOU for 4 points of damage.
[Wed Aug 26 00:02:24 2015] A snake tries to hit YOU, but misses!
[Wed Aug 26 00:02:25 2015] You try to punch a snake, but a snake parries!
[Wed Aug 26 00:02:27 2015] A snake tries to hit YOU, but misses!
[Wed Aug 26 00:02:28 2015] You kick a snake for 1 point of damage.
[Wed Aug 26 00:02:28 2015] You try to punch a snake, but miss!
[Wed Aug 26 00:02:30 2015] A snake hits YOU for 2 points of damage.
[Wed Aug 26 00:02:32 2015] You try to punch a snake, but miss!
[Wed Aug 26 00:02:33 2015] A snake tries to hit YOU, but misses!
[Wed Aug 26 00:02:36 2015] You punch a snake for 5 points of damage.
[Wed Aug 26 00:02:36 2015] A snake hits YOU for 1 point of damage.
[Wed Aug 26 00:02:37 2015] You kick a snake for 1 point of damage.
[Wed Aug 26 00:02:39 2015] A snake hits YOU for 1 point of damage.
[Wed Aug 26 00:02:40 2015] You gain experience!!
[Wed Aug 26 00:02:40 2015] You punch a snake for 6 points of damage.
[Wed Aug 26 00:02:40 2015] You have slain a snake!
[Wed Aug 26 00:02:42 2015] --You have looted a Snake Egg.--
[Wed Aug 26 00:02:42 2015] <SYSTEMWIDE_MESSAGE>: Innoruuk has been defeated by a group of hardy adventurers! Please join us in congratulating Harken along with everyone else who participated in this achievement!
[Wed Aug 26 00:02:43 2015] --You have decided to not loot 1 item(s): Snake Scales. The item(s) will be available to anyone after the corpse(s) unlock.--
[Wed Aug 26 00:02:46 2015] Auto attack is on.
[Wed Aug 26 00:02:46 2015] You punch a large rat for 6 points of damage.
[Wed Aug 26 00:02:46 2015] A large rat tries to bite YOU, but misses!
[Wed Aug 26 00:02:47 2015] You try to kick a large rat, but miss!
[Wed Aug 26 00:02:47 2015] You punch a large rat for 6 points of damage.
[Wed Aug 26 00:02:49 2015] A large rat bites YOU for 1 point of damage.
[Wed Aug 26 00:02:51 2015] You gain experience!!
[Wed Aug 26 00:02:51 2015] You punch a large rat for 5 points of damage.
[Wed Aug 26 00:02:51 2015] You have slain a large rat!
[Wed Aug 26 00:02:53 2015] --You have decided to not loot 1 item(s): Piece of Rat Fur. The item(s) will be available to anyone after the corpse(s) unlock.--
[Wed Aug 26 00:02:53 2015] --You have decided to not loot 2 item(s): Scalded Rat Skin. The item(s) will be available to anyone after the corpse(s) unlock.--
[Wed Aug 26 00:02:56 2015] Auto attack is on.
[Wed Aug 26 00:02:56 2015] You try to punch a large rat, but miss!
[Wed Aug 26 00:02:56 2015] A large rat bites YOU for 1 point of damage.
[Wed Aug 26 00:02:57 2015] You kick a large rat for 1 point of damage.

I have the log file path and name stored in the program's settings like so:

Properties.Settings.Default.setting_logfolder // Folder Path
Properties.Settings.Default.setting_logfile // File Name

I need to open the file when I click a button and read the file till the button is clicked again to stop the process, while the file is being read I need the lines that it finds to be output to a textbox for display to the user. The log file will still be written to by the parent program while this happens.

This is the code for the button that I have so far.

    private void btnStart_Click(object sender, EventArgs e)
    {
        if (btnStart.Text == "Start Parsing")
        {
            // change text on button and switch status image
            btnStart.Text = "Stop Parsing";
            pbStatus.Image = Properties.Resources.online;

            string logfile = Properties.Settings.Default.setting_logfolder += Properties.Settings.Default.setting_logfile;

            if (File.Exists(logfile))
            {
                Stream stream = File.Open(logfile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
                StreamReader streamReader = new StreamReader(stream);
                string str = streamReader.ReadToEnd();
                tbOutput.AppendText(str);
                streamReader.Close();
                stream.Close();
            }
            else
            {
                tbOutput.AppendText(logfile);
            }
        }
        else
        {
            btnStart.Text = "Start Parsing";
            pbStatus.Image = Properties.Resources.offline;
        }
    }

I need it to keep reading the file until the button is pressed again and I only need lines like this one from the file

[Wed Aug 26 00:02:08 2015] <SYSTEMWIDE_MESSAGE>: XXXXXXXXX has been defeated by a group of hardy adventurers! Please join us in congratulating XXXXXX along with everyone else who participated in this achievement!

@kyle - I think it's "give me the regex for `[Wed Aug 26 00:02:08 2015] : XXXXXXXXX has been defeated by a group of hardy adventurers! Please join us in congratulating XXXXXX along with everyone else who participated in this achievement!`... Manvaril - are you sure you want the date-time hard-coded in the regex? — Orphid, Sep 21 '15 at 16:31
A hint: you should not read the whole file to a single variable when the file grows to 30MB+. You might get an out of memory issue. And the whole processing will be very slow. It is safer to read line by line. — Wiktor Stribiżew, Sep 21 '15 at 16:52
If this is for personal use and not for some sort of distribution, I would recommend [Powershell, as it would be a one-liner](http://blogs.technet.com/b/heyscriptingguy/archive/2013/02/24/weekend-scripter-two-way-cool-powershell-text-file-tricks-tail-and-wait.aspx)... if you want help adapting that to filter lines, change your question or ask another one — Code Jockey, Sep 21 '15 at 16:57
`[Wed Aug 26 00:02:08 2015] : XXXXXXXXX has been defeated by a group of hardy adventurers! Please join us in congratulating XXXXXX along with everyone else who participated in this achievement!` I need to pull just the date stamp without [ ] The first XXXXXX Value (can be multple words) and the last XXXXXX Value, How would you do that? — Manvaril, Sep 22 '15 at 17:27

score 1 · Answer 1 · edited May 23 '17 at 11:58

1

My answer "for C#"

To find any line with a system wide message, use:

\[[\w :]+\] <SYSTEMWIDE_MESSAGE>

for exactly that message, but allowing different names, use:

\[[\w :]+\] <SYSTEMWIDE_MESSAGE>: \w+ has been defeated by a group of hardy adventurers! Please join us in congratulating \w+ along with everyone else who participated in this achievement!

These both assume that user/character names can have letters, digits or underscores

Your current algorithm does a ReadToEnd(), which will not read individual lines and cough them up one at a time, which it seems you want to do(?! maybe!!??!). Instead, it will read to the end of the stream (file), then cough up the whole file.

A method using FileSystemWatcher is described briefly in this answer to a related question - there are a few caveats to FileSystemWatcher, but it should be a more "pure .Net approach"

Alternate solution (Powershell)

If you don't need the GUI, and you are merely interested in getting this information put up on screen so you can do something about it, the quick and dirty solution is literally one line in PowerShell land. Open a PowerShell prompt (or create a script) and (assuming we're using a log file humorously named chase.dat and a chosen regex instead of <#regex#>, use the following code:

cat chase.dat -tail 1 -wait | ?{$_ -match '<#regex#>'}

Put an appropriate regex in place of <#regex#> and your file name in place of chase.dat and it will spit out any new lines that match your regex, as they are written to the file. I can explain it if you're interested...

GUI!

If you do want to combine the GUI and PowerShell, the above PS script line would be passed to the "PipelineExecutor" constructor in the class outlined in this article, which would then update the specified property (e.g. TextBox.Text) whenever the script returned more (in this case filtered) text.

Just did this, and it works like a charm!

edited May 23 '17 at 11:58

Community

1
1

answered Sep 21 '15 at 16:46

Code Jockey

6,611
6
33
45

Your second example should work, I'm sorry I wasn't specific in the first post. The timestamp at the beginning of the line will obviously change with time but the message will be the same with the XXXXX in the message being different. I do need to read the file line by line but I don't know how to do that. – Manvaril Sep 21 '15 at 17:17
I am using WPF with a textbox for output – Manvaril Sep 21 '15 at 17:33
If you're comfortable with multi-threading, I'd look at the [top two answers here](http://stackoverflow.com/questions/1409492/read-from-a-growing-file-in-c) (which would probably be the safer/sturdier solution). I'm not the most comfortable with threading (shame on me), so I would probably try to fit my one liner above into [this situation](http://learn-powershell.net/2012/10/14/powershell-and-wpf-writing-data-to-a-ui-from-a-different-runspace/) – Code Jockey Sep 21 '15 at 17:51
There's also [this Code Project post](http://www.codeproject.com/Articles/18229/How-to-run-PowerShell-scripts-from-C) that seems fairly straight forward, and should not require an ugly console window(?) -- not sure – Code Jockey Sep 21 '15 at 17:57
I like the filesystem watcher example given in [the second answer](http://stackoverflow.com/questions/1409492/read-from-a-growing-file-in-c) – Manvaril Sep 21 '15 at 18:09
@Manvaril cool - I can put a reference to that in my answer if you felt it was helpful, though I also found [this method of running Powershell asynchronously via C#](http://www.codeproject.com/Articles/18409/Asynchronously-Execute-PowerShell-Scripts-from-C) that seems it need only be put into WPF, then started with my one-liner -- I think I'll play around with this in the next few days for personal interest. – Code Jockey Sep 21 '15 at 18:13
I don't want to use powershell, I want when someone hits the start button that the file is read line by line looking for that specific line and putting it in the text box, and when they hit the button again that it stops reading the file and is ready to start again when they push the button again. so it needs to read the file continuously till they stop it with the button they started with. – Manvaril Sep 21 '15 at 18:19

Rohit · Accepted Answer · 2015-09-22T05:10:18.430

1

Here is a simple approach using FileWatcher Class

 public static Watchfile() 
 {
         FileSystemWatcher watch = new FileSystemWatcher();
         watch.Path = @"C:\";
         watch.Filter = "log.txt";
         watch.NotifyFilter = NotifyFilters.LastAccess |    
         NotifyFilters.LastWrite; 

        watch.Changed += new FileSystemEventHandler(OnChanged);
       watch.EnableRaisingEvents = true;
  }


 private static void OnChanged(object source, FileSystemEventArgs e)
 {


    string line;
         if (e.FullPath == @"C:\log.txt")
         {
             Regex _regex = new Regex(@"\[[\w :]+\] <SYSTEMWIDE_MESSAGE>: \w+ has been defeated by a group of hardy adventurers! Please join us in congratulating \w+ along with everyone else who participated in this achievement!");
             System.IO.StreamReader file =
                       new System.IO.StreamReader(@"C:\log.txt");

             while ((line = file.ReadLine()) != null)
             {
                 Match match = _regex.Match(line);
                 if (match.Success)
                 {

                     //Match Found

                 }


             }

             file.Close();
         }


 }

edited Sep 22 '15 at 05:10

answered Sep 21 '15 at 17:44

Rohit

10,056
7
50
82

Not sure, but I believe this will simply read the file once, line by line -- I believe the (unasked) question is more or less looking for something to update the WPF Textbox as the file is progressively updated - so it needs to keep running (in another thread?) and updating the text box. – Code Jockey Sep 21 '15 at 18:06
@CodeJockey is correct, it will need to keep running and update, another thread would be best – Manvaril Sep 21 '15 at 18:48
@kyle - that would be necessary, yes, but somewhat wasteful to run constantly. The next missing piece would be something like file system watcher – Code Jockey Sep 21 '15 at 18:54
I'm using your OnChanged function with my own filewatcher setup and when I run a debug I get "Cannot read from a closed TextReader." at this line `while((line = file.ReadLine()) != null)` [pastebin of OnChanged](http://pastebin.com/ZaBy0kAg) – Manvaril Sep 21 '15 at 21:13

Orphid · Answer 3 · 2015-10-27T19:56:56.130

Ok, so first, if you're parsing a log file that might be too large for you to want to hold in memory (not that 20mb is, most of the time), then use File.OpenText and read it in in blocks or lines. This means you don't need to load the whole text file into memory. Secondly, I would remove the reading logic into it's own class or function. If you are running off Windows Forms or WPF and want to update the UI as and when you find a line, then you will need to do this read on a separate thread/task to get your UI to update without locking it. Update your UI from the background task by using the Dispatcher class.

Then, your regex will need to be based around the precise lines you want to capture. It seems to me that you can ignore everything in square braces at the start of a line, and then just assert on:

\w+ has been defeated by a group of hardy adventurers! Please join us in congratulating \w+ along with everyone else who participated in this achievement!$

Edit:

It looks like, from your file sharing options, that the log file might be growing? If this is correct, avoid reading by line as you can read a partial line. If you are caching your position in the stream, this may mean you loose a potential match. Instead, read a block, check for Environment.NewLine, and parse that as a line in your regex engine.

score -1 · Answer 4 · edited May 26 '16 at 14:51

-1

Try this:

-using (var fileStream = new FileStream("YOUR FILE", FileMode.Open, FileAccess.Read, FileShare.None))
{                  
    using (var streamReader = new StreamReader(fileStream))
    {
        string line;
        while((line = reader.ReadLine()) != null)
        {
          if(line == "YOUR TEXT")
          {
           // Do the rest
          }
        }
    }
}

You'll have to specify your specific pattern in order to complete this sample.

edited May 26 '16 at 14:51

Fabian N.

3,807
2
23
46

answered Sep 21 '15 at 17:25

hbulens

1,872
3
24
45

any explanation from the downvoter for voting down my comment? – hbulens Sep 21 '15 at 18:58

Read a Log file and pull out lines with specific text

4 Answers4

My answer "for C#"

Alternate solution (Powershell)