Parse log file and turn it into a custom object

Question

The log file contains:

=====
SMTP_server_not_set_up
@PANEL  SMTP server not set up. Contact system administrator.
@PAGECNT  381
@SCANADFCNT 0
@DATETIME  2019-08-29T10:05:51-0400
@UPTIME  0:00:23
@CODELVL

Base MSNGM.053.023
Engine GM.052.E015
Panel f0.12p48v3
@END
=====
PowerOnReset
@PAGECNT  381
@SCANADFCNT 0
@DATETIME  2019-08-29T10:05:50-0400
@PORCNT  49
@CODELVL

Base MSNGM.053.023
Engine GM.052.E015
Panel f0.12p48v3
@END
=====
Load_MP_Feeder_with_Custom_Type_5_Letter
@PANEL  Load MP Feeder with Custom Type 5 Letter
@PAGECNT  337
@SCANADFCNT 0
@DATETIME  2019-08-29T09:59:22-0400
@UPTIME  0:44:15
@CODELVL

Base MSNGM.053.023
Engine GM.052.E015
Panel f0.12p48v3
@END
=====

... many many more lines

I want to grab the 6 lines following the line "=====" and turn that into a custom object.

For example, based on the first lines, the ideal object would be:

MESSAGE: SMTP_server_not_set_up
PANEL : @PANEL  SMTP server not set up. Contact system administrator.
PAGECNT: @PAGECNT  381
SCANADFCNT :@SCANADFCNT 0
DATETIME : @DATETIME  2019-08-29T10:05:51-0400
UPTIME: @UPTIME  0:00:23

So far I can isolate those 6 lines that follow the recurring "=====" like this:

(Select-String -Path "$PSScriptRoot\$($printer)_history.log" -Pattern "=====" -Context 6).Context.PostContext

Now I need to put those lines in the custom object. I really don't know exactly how to approach this correctly, I'm exploring doing something like this:

$ObjHistory = @()
(Select-String -Path "$PSScriptRoot\$($printer)_history.log" -Pattern "=====" -Context 6).Context.PostContext | ForEach-Object {
    if ($_ -match "PANEL") {
        $properties = [ordered]@{'PANEL'="$_"}
        $objet = New-Object -TypeName PSObject -Property $properties
    }

    $ObjHistory += $objet
}

$ObjHistory

This way I found out I can put that PANEL property in a custom object sucessfully, it outputs:

PANEL
-----
@PANEL  SMTP server not set up. Contact system administrator.
@PANEL  Load MP Feeder with Custom Type 5 Letter
@PANEL  Load MP Feeder with Custom Type 5 Letter
@PANEL  Load MP Feeder with Custom Type 5 Letter
@PANEL  Load MP Feeder with Custom Type 5 Letter

However, I need to do that for the other lines and I'm stuck at how to approach it, since ForEach-Object does one line at a time.

Ansgar Wiechers · Answer 1 · 2019-09-03T16:02:23.627

1

The extracted data already has a key/value kind of structure, so you could use ConvertFrom-StringData to convert the data chunks into hashtables and then into custom objects.

Avoid appending to an array in a loop. Just output the objects in the loop and collect the loop output in a variable.

$path  = "${PSScriptRoot}\${printer}_history.log"

$objHistory = @(Select-String -Path $path -Pattern '=====' -Context 6 | ForEach-Object {
    $props = $_.Context.PostContext -replace '^[^@]', 'MESSAGE = $&' -replace '^@(\w+)', '$1 = $&' |
             Out-String |
             ConvertFrom-StringData
    New-Object -Type PSObject -Property $props
})

-replace '^[^@]', 'MESSAGE = $&' prepends lines that don't begin with a @ with the string MESSAGE =, and -replace '^@(\w+)', '$1 = $&' replaces @WORD at the beginning of a line with WORD = @WORD. The subsequent Out-String then merges the data chunk into a single string, otherwise ConvertFrom-StringData would create one hashtable per line.

edited Sep 03 '19 at 16:02

answered Sep 03 '19 at 15:14

Ansgar Wiechers

193,178
25
254
328

As the above log file itself doesn't have the colon's included (and the 1st line misses the message tag) I get errors with your script. That's why I took a longer road ;-) – Sep 03 '19 at 15:40
@LotPings I mistook the 2nd text snippet for the actual data initially, but had already amended my answer. – Ansgar Wiechers Sep 03 '19 at 15:43
Yes, I see, but the 1st line still throws an error from `ConvertFrom-StringData` ... hasn't the proper Name=Value format – Sep 03 '19 at 15:49
@LotPings Ah, yes. I overlooked the first line. Thanks for the heads up. – Ansgar Wiechers Sep 03 '19 at 16:02

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

A different approach

reading the log in with -raw parameter
splitting at ===== into sections
splitting the sections into lines and storing those lines into variables to output as a [PSCustomObject]

Edit

create an empty [PSCustomObject] and use a RegEx to fill the relevant properties
(cleaning the redundant @propname, if desired)

## Q:\Test\2019\09\03\SO_57773555.ps1
$RE = [regex]'^@(MESSAGE|PANEL|PAGECNT|SCANADFCNT|DATETIME|UPTIME) +(.*)$'
$Path = "$PSScriptRoot\$($printer)_history.log"

$ObjHistory = foreach($Section in ((Get-Content $Path -raw)  -split '===== ?\r?\n' -ne '')){
    $obj = [PSCustomobject]@{MESSAGE="";PANEL="";PAGECNT="";SCANADFCNT="";DATETIME="";UPTIME=""}
    $obj.MESSAGE = ($Section -split '\r?\n')[0]
    foreach($Line in ($Section -split '\r?\n')[1..5] ){
      if($Line -match $RE){
        $obj.($Matches[1])=$Matches[2] # to have the complete line =$Matches[0]
      }
    }
    $obj 
  }
$ObjHistory

> .\SO_57773555.ps1                                                                                                                 

MESSAGE    : SMTP_server_not_set_up
PANEL      : SMTP server not set up. Contact system administrator.
PAGECNT    : 381
SCANADFCNT : 0
DATETIME   : 2019-08-29T10:05:51-0400
UPTIME     : 0:00:23

MESSAGE    : PowerOnReset
PANEL      :
PAGECNT    : 381
SCANADFCNT : 0
DATETIME   : 2019-08-29T10:05:50-0400
UPTIME     :

MESSAGE    : Load_MP_Feeder_with_Custom_Type_5_Letter
PANEL      : Load MP Feeder with Custom Type 5 Letter
PAGECNT    : 337
SCANADFCNT : 0
DATETIME   : 2019-08-29T09:59:22-0400
UPTIME     : 0:44:15

This is truly masterful my friend. Much appreciated. I'm learning a lot just trying to understand each lines. The logic is really genius haha. — Rakha, Sep 03 '19 at 18:41
Quick question to be sure, in the foreach($Section in...), doing "$obj" at the end make it "append" the object each time to $objhistory, right? Therefore combining all the objects into one? — Rakha, Sep 05 '19 at 13:44
Exactly, `$obj` needs to be present with ***all*** properties to be able to stuff the value extracted by the RegEx. — , Sep 05 '19 at 13:48

score 1 · Answer 3 · answered Sep 03 '19 at 16:38

Took a shot at convertfrom-string, example based parsing. Multiple examples work better.

$template = @'
=====
{message*:SMTP_server_not_set_up}
{panel:@PANEL  SMTP server not set up. Contact system administrator.}
{pagecnt:@PAGECNT  381}
{scanadfcnt:@SCANADFCNT 0}
{datetime:@DATETIME  2019-08-29T10:05:51-0400}
{uptime:@UPTIME  0:00:23}
@CODELVL

Base MSNGM.053.023
Engine GM.052.E015
Panel f0.12p48v3
@END
=====
{message*:PowerOnReset}
{pagecnt:@PAGECNT  381}
{scanadfcnt:@SCANADFCNT 0}
{datetime:@DATETIME  2019-08-29T10:05:50-0400}
@PORCNT  49
@CODELVL

Base MSNGM.053.023
Engine GM.052.E015
Panel f0.12p48v3
@END
=====
{message*:Load_MP_Feeder_with_Custom_Type_5_Letter}
{panel:@PANEL  Load MP Feeder with Custom Type 5 Letter}
{pagecnt:@PAGECNT  337}
{scanadfcnt:@SCANADFCNT 0}
{datetime:@DATETIME  2019-08-29T09:59:22-0400}
{uptime:@UPTIME  0:44:15}
@CODELVL

Base MSNGM.053.023
Engine GM.052.E015
Panel f0.12p48v3
@END
=====
'@

$testtext = @'
=====
SMTP_server_not_set_up
@PANEL  SMTP server not set up. Contact system administrator.
@PAGECNT  381
@SCANADFCNT 0
@DATETIME  2019-08-29T10:05:51-0400
@UPTIME  0:00:23
@CODELVL

Base MSNGM.053.023
Engine GM.052.E015
Panel f0.12p48v3
@END
=====
PowerOnReset
@PAGECNT  381
@SCANADFCNT 0
@DATETIME  2019-08-29T10:05:50-0400
@PORCNT  49
@CODELVL

Base MSNGM.053.023
Engine GM.052.E015
Panel f0.12p48v3
@END
=====
Load_MP_Feeder_with_Custom_Type_5_Letter
@PANEL  Load MP Feeder with Custom Type 5 Letter
@PAGECNT  337
@SCANADFCNT 0
@DATETIME  2019-08-29T09:59:22-0400
@UPTIME  0:44:15
@CODELVL

Base MSNGM.053.023
Engine GM.052.E015
Panel f0.12p48v3
@END
=====
'@

$testText | ConvertFrom-String -TemplateContent $template

Output:

message    : SMTP_server_not_set_up
panel      : @PANEL  SMTP server not set up. Contact system administrator.
pagecnt    : @PAGECNT  381
scanadfcnt : @SCANADFCNT 0
datetime   : @DATETIME  2019-08-29T10:05:51-0400
uptime     : @UPTIME  0:00:23

message    : PowerOnReset
pagecnt    : @PAGECNT  381
scanadfcnt : @SCANADFCNT 0
datetime   : @DATETIME  2019-08-29T10:05:50-0400

message    : Load_MP_Feeder_with_Custom_Type_5_Letter
panel      : @PANEL  Load MP Feeder with Custom Type 5 Letter
pagecnt    : @PAGECNT  337
scanadfcnt : @SCANADFCNT 0
datetime   : @DATETIME  2019-08-29T09:59:22-0400
uptime     : @UPTIME  0:44:15

To my experience `ConvertFrom-String` isn't very reliable due to the heuristics. The better/more detailed you describe the data the more unpredictable errors occur, just when you don't expect them. — , Sep 03 '19 at 17:12
Even though this seems this isn't the optimal way to solve the problem, I'm very interested to discover the template / convertfrom-string method described. Thanks — Rakha, Sep 03 '19 at 19:19

Parse log file and turn it into a custom object

3 Answers3

Edit