Grok Parse Failure on Custom Log Format and regex in logstash

Question

I have a custom log format ,i am new to it so trying to figure out how it works . It is not getting parsed in logstash .Can someone help to identify the issue.

Logformat is as follows

{u'key_id': u'1sdfasdfvaa/sd456dfdffas/zasder==', u'type': u'AUDIO'}, {u'key_id': u'iu-dsfaz+ka/q1sdfQ==', u'type': u'HD'}], u'model': u'Level1', u'license_metadata': {u'license_type': u'STREAMING THE SET', u'request_type': u'NEW', u'content_id': u'AAAA='}, u'message_type': u'LICENSE', u'cert_serial_number': u'AAAASSSSEERRTTYUUIIOOOasa='}

I need to get it parsed in logstash and then store it in elasticsearch

The problem is the none of the existing grok pattern are taking care of it and i am unaware of regex custom config

Answer 1

Alain's comment may be useful to you, if that log is, in fact, coming in as JSON you may want to look at the JSON Filter to automajically parse a JSON message into an elastic friendly format or using the JSON Codec in your input.

If you want to stick with grok, a great resource for building custom grok patterns is Grok Constructor.

Answer 2

It seems like you're dumping a json hash from python 2.x to a logfile, and then trying to parse it from logstash.

First - Fix your json format and encoding: Your file doesn't correclty generated json strings. My recommendation is to fix it on your application before trying to consume the data from Logstash, if not you'll have to make use of some tricks to do it from there:

 # Disable accii default charset and encode to UTF-8
js_string = json.dumps(u"someCharactersHere", ensure_ascii=False).encode('utf8')

 # validate that your new string is correct
 print js_string

Second - Use the Logstash JSON filter

Grok is module intended to parse any kind of text using regular expressions. Every expression converts to a variable, and those variable can be converted to event fields. You could do it, but it will be much more complex and prune to errors.

Your input has a format already (json), so you can make use of Logstash JSON Filter. It will do all the heavy lifting for you by converting the json structure into fields:

filter {
  json {
    # this is your default input. you shouldn't need to touch it
    source => "message"

    # you can map the result into a variable. Simply uncomment the
    # following:
    # target => "doc"


    # note: if you don't use the target option. the filter will try to 
    # map the json string into fields into the 'root' of your event
  }
}

Hope it helps,