1

I have some costum log files I would like to parse so I can feed them in logstash. I built a grok pattern to parse them but admitedly I'm not very proficient in grock nor regex, I wanted to ask if what I did could somehow be simplified/optimized.

Logs example:

  system_info : Calculator[1]_Global@HKGL1V5KY33 (23572.0000000007A668B0) : [2020/10/14-15:43:17.975] : GridTaskProcessor::mainThreadRun() : Routing criteria : GridServiceName = SophisMonteCarlo
  system_warning : Calculator[1]_Global@HKGL1V5KY33 (23572.0000000007A668B0) : [2020/10/14-15:43:07.840] :vDBFees::loadFeesDetailsMapping() : Broker detail fees mapping begin
  system_info : NamingServer[standard]@BMALAN (10276.000000001C8DA890) : [2020/08/05-15:04:13.426] : GenericServiceLoader::InitGenericServices() : Initializing generic service 'MonitoringHubConnector'...
  system_info : ServerAdminConsole[standard]@BMALAN (8880.000000001B7A75D0) : [2019/05/31-15:04:23.240] : ServerAdminConsole::backgroundWorker_DoWork() : Initializing Entries from naming service....

Grok Pattern:

\s*(?<verbosity>(.*?)(?=\ :))\s*:\s*(?<servicename>(.*?)(?=\@))@(?<servername>(.*?)(?=\ ))\s*\((?<threadid>(.*?)(?=\)))\)\s*:\s*\[(?<date>(.*?)(?=\-))\-(?<time>(.*?)(?=\]))\]\s*:\s*(?<class>(.*?)(?=\())\(\)\s*:\s*%{GREEDYDATA:message}

Grok Parsing Explained

Benou
  • 11
  • 1

1 Answers1

0

Well generally spoken, you don't necessarily need to do all the lookaheads for every part of the log message (=line). Positive Lookaheads are causing more processing time. This does not mean that lookaheads are bad but you should ask yourself if you really need them.

In my opinion, positive lookaheads are useful if you are not sure if the part/character of the log line occurrs.

Many logs follow a defined pattern that is somewhere documented (e.g. the Apache Access Log).

Parts of the log line that not always occurr can then be set optional with the ?-quantifier like

.*(%{DATA:some_field})?.*

Furthermore, events that did not match the grok pattern get the tag _grokparsefailure (default). You can search for events with this tag in Kibana, analyze why they did not match and then adapt your pattern.

So to sum it up:

Ask yourself if you can rely on a basic pattern and dismiss the lookaheads.

I hope I could help you.

apt-get_install_skill
  • 2,818
  • 10
  • 27