0

I need a help for writing the filters for logstash. My logstash is configured to read syslog.

The log message is as follows,

Mar 14 15:11:11 localhost 192.168.235.136 {'status': 'True', 'endpoint': '/search/basic/', 'parameters': <QueryDict: {u'fileName': [u'Adware']}>, 'company': u'Global first', 'matched threat scape': [u'Enterprise IT Management and Investment'], 'request id': 11, 'user id': 2L, 'user': u' ', 'matched report id': [u'Intel-732102']}

I wanted to have filters on kibana, based on the json keys that I am passing in the message to logstash.

I am not able to write the filters to get the parameters from my logs. I have also tried http://grokdebug.herokuapp.com/ for generating the filter. It gave me the pattern that I am not sure how to use it.

{%{QS:'status'}: %{QS}, %{QS}: %{QS}, %{QS}: <QueryDict: {u%{QS}: %{SYSLOG5424SD}}>, %{QS}: u%{QS}, %{QS}: %{SYSLOG5424SD}, %{QS}: 11, %{QS}: 2L, %{QS}: u' ', 'matched report id': %{SYSLOG5424SD}}
Bhupesh Pant
  • 4,053
  • 5
  • 45
  • 70

1 Answers1

1

For the log input:

Mar 14 15:11:11 localhost 192.168.235.136 {'status': 'True', 'endpoint': '/search/basic/'}

Grok Pattern is

%{CISCOTIMESTAMP:JsonTimestamp} localhost %{IP:JsonIP} {'status': '%{WORD:JsonStatus}', 'endpoint': '%{UNIXPATH:JsonPath}'}

Please follow similar pattern approach for the rest of the fields. You can find the references at:

https://github.com/elasticsearch/logstash/blob/master/patterns/grok-patterns

Please post what fields you feel difficulty in applying grok pattern, along with following details:

i) What fields are varying and what are constants.

In the above example JsonTimestamp,JsonIP,JsonStatus and JsonPath will be indexed.

Please find below working full grok pattern for above example:

%{CISCOTIMESTAMP} localhost %{IP} {%{QS}: %{QS}, %{QS}: %{QS}, %{QS}: <QueryDict: {u%{QS}: %{SYSLOG5424SD}}>, %{QS}: u%{QS}, %{QS}: %{SYSLOG5424SD:matched_threat_scape}, %{QS}: %{NUMBER:request_id}, %{QS}: %{NUMBER:user_id}L, %{QS}: %{WORD:user}%{QS}, %{QS}: %{SYSLOG5424SD:matched_report_id}

modified changes are %{NUMBER:request_id}, %{NUMBER:user_id}L, %{WORD:user}, %{SYSLOG5424SD:matched_report_id} since 11 and 2L are not in quotes and they represent numbers, we use NUMBER to identify numeric tokens, user is represented as WORD token

Murthy
  • 377
  • 2
  • 10
  • Thanks for your reply Murthy. I managed to parse the data in the similar way you mentioned. But still I am not able to parse complete data. I am getting the error when I try to parse the complete message.. – Bhupesh Pant Mar 18 '14 at 14:16
  • filter { if [status] == 'True' { grok { add_tag => [status, end_point, params_name, params, company, matched_threat_scape, request_id, user_id, syslog_message] type => "my_audit" match => [ "message", "%{CISCOTIMESTAMP} localhost %{IP} {%{QS}: %{QS}, %{QS}: %{QS}, %{QS}: , %{QS}: u%{QS}, %{QS}: %{SYSLOG5424SD:matched_threat_scape}, %{QS}: %{QS:request_id}, %{QS}: %{QS:user_id}, %{GREEDYDATA:syslog_message}}" ] } } } – Bhupesh Pant Mar 18 '14 at 14:17
  • Please find the working full grok pattern in last lines of above edited answer – Murthy Mar 19 '14 at 05:41
  • It still does not include last two values parsed, user': u' ', 'matched report id': [u'Intel-732102'] – Bhupesh Pant Mar 19 '14 at 07:00
  • Please find updated grok pattern. You can find pattern references at: https://github.com/elasticsearch/logstash/blob/master/patterns/grok-patterns – Murthy Mar 19 '14 at 08:16
  • Actually my problem is that even if I am using the correct pattern for the extracting the values, I am not able to fetch it. I am getting the error the and if I am using GREEDYDATA:syslog_message in for the last message then my problem is solved.. – Bhupesh Pant Mar 19 '14 at 12:30
  • i)Did you try with latest pattern in answer at http://grokdebug.herokuapp.com/, i don't see any exceptions and fetched request_id, user_id, user, matched_report_id tokens. ii) Apart from request_id, user_id, user, matched_report_id what fields u require? – Murthy Mar 20 '14 at 05:24