logstash Grok to extract different data from log file containing different log

Question

My log file contains data from different process writing data on same file. The log file is something like as shown below.

I am writing to write the Grok filter pattern to extract different data and use it in Kibana board. I tried one pattern but it only works for one of the line in log file, it does not work for the whole log file.

%{UUID:uuid} > %{WORD:level}:%{INT:LOGlevel} %{WORD:RClevel}: %{INT:RClevel} %{WORD:LOGtype} :%{GREEDYDATA:message}

I need data on MGMT_RDCIP_INFO, PCI, DP_DRIVER from the log such as RATIO, QUALITY, Ceiling data. Can anyone guide me how do I grab specific keyword data from the log.

ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0 MGMT_RDCIP_INFO :Bandwidth Management for Server: Ceiling = 112500.000000, Floor = 12500.000000, Active = 14825.552639
ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0             PCI :Display 0 codec 0 (H264 Encoder) frames encoded per second : 11.56
ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0             PCI :Display 1 codec 0 (H264 Encoder) frames encoded per second : 25.92
ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0             PCI :QUALITY: 81.3918 81.3918 0.0 0.0
ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0             PCI :RATIO: 5.73013 94.2699 0.0 0.0
ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:3 CT:   0       DP_DRIVER :Display duplication output id: 1 move MPPS 0.00, dirty MPPS 162.59, total MPPS 162.59```

[![snaphot of log file][1]][1]



  [1]: https://i.stack.imgur.com/wuFum.jpg

YouryDW · Answer 1 · 2021-04-29T04:55:08.203

A UUID can best be seen as the DATA type, also make sure you are not picking up the spaces in front of the RCLevel and the LOGtype

%{DATA:uuid} > %{WORD:LOGlevel_WORD}:%{INT:LOGlevel_INT} %{WORD:RClevel_WORD}:[ ]{0,99}%{NUMBER:RClevel_NUMBER}[ ]{0,99}%{WORD:LOGtype} :%{GREEDYDATA:message}

This gives me an output like:

{
  "RClevel_INT": "0",
  "LOGlevel_WORD": "LVL",
  "LOGtype": "MGMT_RDCIP_INFO",
  "RClevel_WORD": "CT",
  "LOGlevel_INT": "3",
  "message": "Bandwidth Management for Server: Ceiling = 112500.000000, Floor = 12500.000000, Active = 14825.552639",
  "uuid": "ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff"
}

{
  "RClevel_INT": "0",
  "LOGlevel_WORD": "LVL",
  "LOGtype": "PCI",
  "RClevel_WORD": "CT",
  "LOGlevel_INT": "3",
  "message": "Display 0 codec 0 (H264 Encoder) frames encoded per second : 11.56\r",
  "uuid": "ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff"
}

Update (2021-04-29): there can be negative CT values without a space in front of them, updated the grok

Thank you that helped a lot. Is there was to extract specific data from message part, like frames encoded per second. Also the CT values sometime appears negative integer such as `ef22119900-99ecf-10e9-7dfc-ffe0ea066dfff > LVL:1 CT:-520 PCI :No codec ID meta data on pixel source` So I changed %{INT:RClevel_INT} to %{GREEDYDATA:ctmessage}. — Rajendra V, Apr 29 '21 at 03:17
remember that you can use the NUMBER data type also and that you can change your pattern to use a count with a minimum and maximum amount of spaces, will update the snippet for this — YouryDW, Apr 29 '21 at 04:52

yuliansen · Accepted Answer · 2021-05-04T04:34:03.353

it seems like the problem is when you use a single space as the delimiter right?

logstash got grok %{SPACE} it will remove the whole space until the next character

my grok filter

%{UUID:uuid} > %{WORD:level}:%{INT:LOGlevel} %{WORD:RClevel}:%{SPACE}%{WORD:LOGtype}%{SPACE}%{DATA:stuffyouwant}%{SPACE}:%{GREEDYDATA:message}

I've tested it and it works on all of the case.

edit

Seems like you have case that can be useful using if statement. It need 2 grok as

filter{
    grok{
        match{
            "message"="%{UUID:uuid} > %{WORD:level}:%{INT:LOGlevel} %{WORD:RClevel}:%{SPACE}%{WORD:LOGtype}%{SPACE}%{DATA:stuffyouwant}%{SPACE}:%{DATA:parameters}: %{GREEDYDATA:stuffs}"
        }
    }
    if [parameters] == "RATIO"{
        grok{
            match{
                "stuff"="%{NUMBER:ratio1} %{NUMBER:ratio2} %{NUMBER:ratio3} %{NUMBER:ratio4}%{GREEDYDATA:allratio}"
            }
        }
    } else if [parameters]=="QUALITY"{
        grok{
            match{"stuff"="%{NUMBER:q1} %{NUMBER:q2} %{NUMBER:q3} %{NUMBER:q4}%{GREEDYDATA:allq}"
            }
        } 
    }else if [parameters]==""{
        grok{
            etc...
        }
    }
    }
}

first grok to identify parameters, and second grok on each of if statements get the number based on character you need

Thank you. My main goal is to get the information from the GREEDYDATA. information such as RATIO, QUALITY, Ceiling etc to plot the graph. Any idea how do I grab specific keyword data from the log — Rajendra V, May 03 '21 at 18:47
pardon my misconception, I've added some sample filter that seems to suit your needs, if if my answer help you please kindly upvote it or if it fulfill your needs please kindly accept it as answer. — yuliansen, May 04 '21 at 04:35

logstash Grok to extract different data from log file containing different log

2 Answers2

edit