Multiple patterns in one log

Question

So I wrote now several patterns for logs which are working. The thing is now, that I have these multiple logs, with multiple patterns, in one single file. How does logstash know what kind of pattern it has to use for which line in the log? ( I am using grok for my filtering ) And if you guys would be super kind, could you give me the link to the docs, because I weren't able to find anything regarding this :/

score 22 · Accepted Answer · answered Feb 11 '15 at 11:37

You could use multiple patterns for your grok filter,

grok {
  match => ["fieldname", "pattern1", "pattern2", ..., "patternN"]
}

and they will be applied in order but a) it's not the best option performance-wise and b) you probably want to treat different types of logs differently anyway, so I suggest you use conditionals based on the type or tags of a message:

if [type] == "syslog" {
  grok {
    match => ["message", "your syslog pattern"]
  }
}

Set the type in the input plugin.

The documentation for the currently released version of Logstash is at http://logstash.net/docs/1.4.2/. It probably doesn't address your question specifically but it can be inferred.

BornToCode · Answer 2 · 2017-05-29T15:49:42.777

17

Write the most specific grok first and use this syntax:

grok {
    match => {
      "message" => [
      #Most specific grok:
        "%{TIMESTAMP_ISO8601:temp_date}%{SPACE}%{LOGLEVEL:log_level}%{UUID:user_id}",
      #Less specific:
        "%{TIMESTAMP_ISO8601:temp_date}%{SPACE}%{GREEDYDATA:log_message}"
     ]
  }
}

edited May 29 '17 at 15:49

answered May 29 '17 at 15:09

BornToCode

9,495
9
66
83

Can the constituent patterns also be defined in this way, or do you have to use regex alternations (`|`)? – Alexander Apr 09 '18 at 21:19
@Alexander - I'm not sure what you mean by the 'constituent patterns'? – BornToCode Apr 10 '18 at 08:08
You can write your own patterns, just like the built in ones like `SPACE`, `LOGLEVEL`, `UUID`, etc. They're expressed as regular expressions, but my issue with that is that having too many alternations in the regex makes it a really long 1-liner. I'm wondering if there is a similar pattern than can be used to define a pattern in terms of an array of regexes, each one tried in turn, just like how the patterns for `message` are defined here – Alexander Apr 10 '18 at 15:46
@Alexander - Oh now I understand. Well I didn't bump into such feature, but it might be worth to open a new question for this – BornToCode Apr 11 '18 at 08:51

Multiple patterns in one log

2 Answers2

Linked