0

I created a filter to break apart our log files and am having the following issue. I'm not able to figure out how to save the parts of the "message" to their own field or tag or whatever you call it. I'm 3 days new to logstash and have had zero luck with finding someone here who knows it.

So for an example lets say this is your log line in a log file

2017-12-05 [user:edjm1971] msg:This is a message from the system.

And what you want to do is to get the value of the user and set that into some index mapping so you can search for all logs that were by that user. Also, you should see the information from the message in their own fields in Kibana.

My pipeline.conf file for logstash is like

grok {
  match => {
    "message" => "%{TIMESTAMP_ISO8601:timestamp} [sid:%{USERNAME:sid} msg:%{DATA:message}"
  }
  add_tag => [ "foo_tag", "some_user_value_from_sid_above" ]
}

Now when I run the logger to create logs data gets over to ES and I can see the data in KIBANA but I don't see foo_tag at all with the sid value. How exactly do I use this to create the new tag that gets stored into ES so I can see the data I want from the message?

Note: using regex tools it all appears to parse the log formats fine and the log for logstash does not spit out errors when processing.

Also for the logstash mapping it is using some auto defined mapping as the path value is nil.

I'm not clear on how to create a mapping for this either.

Guidance is greatly appreciated.

edjm
  • 4,830
  • 7
  • 36
  • 65
  • It seems like your grok filter is not correct, could you try something like `%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} \[user:%{USERNAME:sid}\] msg:%{GREEDYDATA:yourmessage}` and see if you can filter for sid in Kibana? – Phonolog Dec 09 '17 at 13:11
  • I was under the impression that if errors are not thrown in the logs then the syntax is fine. Guess I'll need to get the debugger installed. – edjm Dec 10 '17 at 14:06
  • Yeah alternativly you can test your grok patterns online on https://grokconstructor.appspot.com/do/match. Newer Kibana versions (I think starting with 5.6?) also [have a debugger included](https://www.elastic.co/guide/en/kibana/current/grokdebugger-getting-started.html) – Phonolog Dec 10 '17 at 14:47
  • I tracked down the issue to the regular expression part itself. Meaning not the Grok pre-built item but something as basic as doing a ([a-z]+) fails. Can you give me an example of how the heck to write this for Grok to understand? – edjm Dec 11 '17 at 18:36
  • Hmm you can checkout the supported grok expression provided by logstash [here](https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns). Hope this helps... – Phonolog Dec 12 '17 at 08:25
  • Another question. I've gotten it all to work but there is one thing that I cannot figure out. How do you specify the character [ in the match section? Only when I create a customized regex can I get that to work, but that just seems friggin stupid to have to do that to match a character e.g. %{OPENBRACKET : "\\[" } – edjm Dec 12 '17 at 13:08
  • 1
    Escacping brackets with a backslash like so `\[` should work everywhere. See one of my answers [here](https://stackoverflow.com/questions/45559913/need-help-in-writing-the-grok-pattern/45563207#45563207) f.e. – Phonolog Dec 12 '17 at 14:14
  • honestly i've been trying that for the past day and the debugger kept saying syntax error when I added it in. Just tried again and it worked. Thanks! – edjm Dec 12 '17 at 14:22

0 Answers0