Powershell - how to handle non timestamped log file entries, including blank lines

Question

I asked a question about how to extract data from a log file based on a start and end timestamp, recently which I got a great and quick answer for, see this question / post for details.

Powershell - How to select a block of text from a text based log file that has a time stamp in the log file entry

However, I now find I have an issue in what I am trying to acheive as the log files I am handling / querying do not have a time stamp entry on every line, and in fact some lines are blank and there are also XML entries that are split across multiple lines.

A brief santised extract is shown here to try and show an idea of the log file data I am trying to handle:

1600 00:06:45 CMD1: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
OUTBOUND XML STRING RESULT OF TRANSFORMATION:

<?xml version="1.0"?>

<TgwGiMessage version="1.0">

    <ShortDataTransmitRequest>

        <SourceITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        <DestinationAddress>

            <DestinationITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        </DestinationAddress>

        <DeliveryConfirmation>1</DeliveryConfirmation>

        <MessageText>XML Text Message</MessageText>

    </ShortDataTransmitRequest>

</TgwGiMessage>


16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
2138 00:06:45 01-BASICDT::HS: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
1e0c 00:06:45 IOM WRITE: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
APD OUTBOUND XML STRING RESULT OF TRANSFORMATION:

The previous question / post has given me the solution I need to be able to identify and output data for the log files so long as the log file entry has a time stamp.

If the log file entry doesn't have a time stamp I get errors reported when the script tries to process these lines:

Index was outside the bounds of the array.
At line:9 char:5
+ if ($parts[1] -ge $StartTime -and $parts[1] -le $EndTime) {
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], IndexOutOfRangeException
    + FullyQualifiedErrorId : System.IndexOutOfRangeException

This is obvioulsy due to the fact that the blank lines can't be split, and those without a timestamp, even if they can be split don't meet my requirement for testing as timestamps.

Could someone suggest how I could approach reading through the log files and outputing all entries from the log file between 2 time stamps.

I'm not a coder, just an admin hacker really so I am struggling to workout how to approach the issue.

I'm more than happy to give anythng a try, but any input to give me an idea of how best to try and tackle this would be most appreciated.

UPDATE - PSGuy, following your last update I have played around a little to try and ascertain why I am not getting the required result when applying your script to my live logfile.

I have created a slightly longer log file example to run against, ExampleLogfile.log as shown below, this gives a few lines where I am trying to extract the exact content of the log, including the XML lines.

1600 00:06:45 CMD1: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:06:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:07:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:07:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:07:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:07:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:08:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:08:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
OUTBOUND XML STRING RESULT OF TRANSFORMATION:

<?xml version="1.0"?>

<TgwGiMessage version="1.0">

    <ShortDataTransmitRequest>

        <SourceITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        <DestinationAddress>

            <DestinationITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        </DestinationAddress>

        <DeliveryConfirmation>1</DeliveryConfirmation>

        <MessageText>XML Text Message</MessageText>

    </ShortDataTransmitRequest>

</TgwGiMessage>


16e8 00:08:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
2138 00:08:45 01-BASICDT::HS: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:09:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
1e0c 00:09:45 IOM WRITE: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:09:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:09:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:10:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
APD OUTBOUND XML STRING RESULT OF TRANSFORMATION:
1600 00:10:45 CMD1: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:10:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:10:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
OUTBOUND XML STRING RESULT OF TRANSFORMATION:

<?xml version="1.0"?>

<TgwGiMessage version="1.0">

    <ShortDataTransmitRequest>

        <SourceITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        <DestinationAddress>

            <DestinationITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        </DestinationAddress>

        <DeliveryConfirmation>1</DeliveryConfirmation>

        <MessageText>XML Text Message</MessageText>

    </ShortDataTransmitRequest>

</TgwGiMessage>


16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
2138 00:13:45 01-BASICDT::HS: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:13:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
1e0c 00:13:45 IOM WRITE: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:13:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:14:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:14:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
APD OUTBOUND XML STRING RESULT OF TRANSFORMATION:
1600 00:14:45 CMD1: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:14:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:15:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:15:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:15:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:15:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:16:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:16:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:16:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:16:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
OUTBOUND XML STRING RESULT OF TRANSFORMATION:

<?xml version="1.0"?>

<TgwGiMessage version="1.0">

    <ShortDataTransmitRequest>

        <SourceITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        <DestinationAddress>

            <DestinationITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        </DestinationAddress>

        <DeliveryConfirmation>1</DeliveryConfirmation>

        <MessageText>XML Text Message</MessageText>

    </ShortDataTransmitRequest>

</TgwGiMessage>


16e8 00:17:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
2138 00:17:45 01-BASICDT::HS: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:17:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
1e0c 00:17:45 IOM WRITE: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:18:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:18:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:18:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
APD OUTBOUND XML STRING RESULT OF TRANSFORMATION:

I have then edited your code as below, to extract a 5 minute section of log from 00:08:45

$file = Get-Content "c:\temp\ExampleLogfile.log"

# create your text pattern for regex matches here
$myPattern = "\d{1,2}\:\d{1,2}\:\d{1,2}"

# what time boundaries do you want to check?
$tempTime = "00:08:45"
$lowerBound = [DateTime]::Parse($tempTime)
$upperBound = $lowerBound.AddMinutes(5)
    # tempTime can use any System.DateTime static method once
    # you have a date time, and should ideally be an input parameter
    # lowerBound should also be an input parameter

# this could also be a for, do / while, or for each
# loop through the file until we reach the end
for ($i = 0; $i -le $file.GetUpperBound(""); $i++)
{
    # this will loop until we reach the last line
    if ($file[$i] -match $myPattern)
    {
        $time = $file[$i].Split(' ')[1]
        if ([DateTime]::Parse($time) -ge $lowerBound -and [DateTime]::Parse($time) -le $upperBound)
        {
            $file[$i] # this will print to a console
                # using the Tee-Object cmdlet will output it to the console
                # and a file, which could be useful for developers and you
        }
    }
}

Which produced the following output:

16e8 00:08:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:08:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:08:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
2138 00:08:45 01-BASICDT::HS: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:09:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
1e0c 00:09:45 IOM WRITE: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:09:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:09:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:10:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
1600 00:10:45 CMD1: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:10:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:10:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
2138 00:13:45 01-BASICDT::HS: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:13:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:
1e0c 00:13:45 IOM WRITE: Standard log file entry: Standard log file entry: Standard log file entry:
16e8 00:13:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:

So it gave me all the timestamped lines from the start time ($tempTime = "00:08:45") for the number of minutes ($upperBound = $lowerBound.AddMinutes(5)) but it did not output the XML or other non timestamped lines. When using the above ExampleLogfile.log I was trying to acheive the following being output:

16e8 00:08:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:08:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
OUTBOUND XML STRING RESULT OF TRANSFORMATION:

<?xml version="1.0"?>

<TgwGiMessage version="1.0">

    <ShortDataTransmitRequest>

        <SourceITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        <DestinationAddress>

            <DestinationITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        </DestinationAddress>

        <DeliveryConfirmation>1</DeliveryConfirmation>

        <MessageText>XML Text Message</MessageText>

    </ShortDataTransmitRequest>

</TgwGiMessage>


16e8 00:08:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
2138 00:08:45 01-BASICDT::HS: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:09:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
1e0c 00:09:45 IOM WRITE: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:09:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:09:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:10:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
APD OUTBOUND XML STRING RESULT OF TRANSFORMATION:
1600 00:10:45 CMD1: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:10:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:10:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:11:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
OUTBOUND XML STRING RESULT OF TRANSFORMATION:

<?xml version="1.0"?>

<TgwGiMessage version="1.0">

    <ShortDataTransmitRequest>

        <SourceITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        <DestinationAddress>

            <DestinationITSI SSI="XXXXXXX" EXT="0" CPTI="0" SNA="0"/>

        </DestinationAddress>

        <DeliveryConfirmation>1</DeliveryConfirmation>

        <MessageText>XML Text Message</MessageText>

    </ShortDataTransmitRequest>

</TgwGiMessage>


16e8 00:12:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
2138 00:13:45 01-BASICDT::HS: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:13:45 CL: Standard log file entry: Standard log file entry: Standard log file entry: 
1e0c 00:13:45 IOM WRITE: Standard log file entry: Standard log file entry: Standard log file entry: 
16e8 00:13:45 CL: Standard log file entry: Standard log file entry: Standard log file entry:

Am I doing something wrong here? This sounds like a simple requirement but I'm struggling with this one.

PSGuy · Answer 1 · 2014-09-24T23:33:34.383

I would try something like this:

$file = Get-Content MyFile.log

# create your text pattern for regex matches here
$myPattern = "\d{1,2}\:\d{1,2}\:\d{1,2}"

# what time boundaries do you want to check?
$tempTime = "00:06:40"
$lowerBound = [DateTime]::Parse($lowerBound)
$upperBound = $lowerBound.AddMinutes(30)
    # tempTime can use any System.DateTime static method once
    # you have a date time, and should ideally be an input parameter
    # lowerBound should also be an input parameter

# this could also be a for, do / while, or for each
# loop through the file until we reach the end
for ($i = 0; $i -le $file.GetUpperBound(""); $i++)
{
    # this will loop until we reach the last line
    if ($file[$i] -match $myPattern)
    {
        $time = $file[$i].Split(' ')[1]
        if ([DateTime]::Parse($time) -ge $lowerBound -and [DateTime]::Parse($time) -le $upperBound)
        {
            $file[$i] # this will print to a console
                # using the Tee-Object cmdlet will output it to the console
                # and a file, which could be useful for developers and you
        }
    }
}

I've also added this to pastebin for you. What the script is doing here is pretty simple, but I'll explain a little more

Get the file content (this comes in as an array of strings by default)
Create a pattern that will match the time format of your file
Declare the time that you want (tempTime, this would be your input param)
Parse the time and obtain the upperBound time (30 mins, 60 mins, etc.)
step through each line of the file, and if it matches the regex, then split on each space, grab array index 1 (the one it will be at), and store it in the time variable
Determine if the time (which is still in string format) is greater than or equal to your lower time boundary and less than or equal to your upper time boundary
Print the lines that match
Return

If you add a variable in here of type System.String[] (do this with the line $myVariable = @() ), then you can do $myVariable += $file[$I], and then use the return keyword at the end of the function that encapsulates this script here.

Does this help provide a comprehensive answer? There are certainly problems with this code for other formats, but because we're purposing it explicitly for the log format you have, it should work just fine.

It's on pastebin at: http://pastebin.com/p1k7XPpD

Also, I'm sorry I didn't get a chance to answer last night. I was having a computer issue and fell asleep trying to resolve it.

Update: --------------------------------- One other thing: I see you were using the string itself in the last thread, and just doing a simple string comparison. That would work, if you wanted to just go that route. Creating the two DateTime objects isn't a requirement. To load all of the code above into a function, you'd have two input parameters: One of type System.String, and another of type System.UInt32 (the difference between Int32 and UInt32 is that UInt32 doesn't allow negative numbers). Your function would then be called like this:

Get-LogExtract -StartTime "00:06:40" -AddMinutes 30

You'd just need to add a reference to AddMinutes in your function. The alternative would be modify what I've given you here, and simply parse both of them with the DateTime.Parse method. Or, alternatively, you can keep it as a string comparison; I'm using DateTime, but there wouldn't be anything wrong with using a string if you know it works. I ran this against your sample output, and it was great.

I just saw what the problem is re-reading the code. I changed a few variable names around after getting it working, because they were somewhat ambiguous previously. If you change line 8 ($lowerBound = [DateTime]::Parse($lowerBound) ) to: $lowerBound = [DateTime]::Parse($tempTime), this will fix the script. Sorry about that! — PSGuy, Sep 26 '14 at 02:45

score 0 · Answer 2 · answered Sep 28 '14 at 03:27

I updated the code having seen the error above... Not sure if you saw the comment, so here it is again:

$file = Get-Content MyFile.log

# create your text pattern for regex matches here
$myPattern = "\d{1,2}\:\d{1,2}\:\d{1,2}"

# what time boundaries do you want to check?
$tempTime = "00:06:40"
$lowerBound = [DateTime]::Parse($tempTime)
$upperBound = $lowerBound.AddMinutes(30)
    # tempTime can use any System.DateTime static method once
    # you have a date time, and should ideally be an input parameter
    # lowerBound should also be an input parameter

# this could also be a for, do / while, or for each
# loop through the file until we reach the end
for ($i = 0; $i -le $file.GetUpperBound(""); $i++)
{
    # this will loop until we reach the last line
    if ($file[$i] -match $myPattern)
    {
        $time = $file[$i].Split(' ')[1]
        if ([DateTime]::Parse($time) -ge $lowerBound -and [DateTime]::Parse($time) -le $upperBound)
        {
            $file[$i] # this will print to a console
                # using the Tee-Object cmdlet will output it to the console
                # and a file, which could be useful for developers and you
        }
    }
}

PSGuy - thanks for the update. I had seen your post, but didn't have chance to reply. I had also tried the change you have advised here when trying to troubleshoot the script assuming it was something basic, but even with the code you supplied here, changing just the variable for the log file name, the start time ($tempTime) and the $upperBound = $lowerBound.AddMinutes(2), the script still starts to output the log file content from 00:00:00. Thanks for continuing to try and assist on this one, Andy — Andy Carroll, Sep 29 '14 at 09:18
PSGuy - I have updated my original question with some further details to show current progress. I have created a short test file ExampleLogfile.log to try and get the script working, but I seem to be going backwards now... see the UPDATE for details. Andy — Andy Carroll, Sep 29 '14 at 10:23

Powershell - how to handle non timestamped log file entries, including blank lines

2 Answers2

Linked