writing grok pattern to split key value pairs

Question

This is my first experience with using grok and elasticsearch. I'm trying to write a grok file to do parse the following line.

2017-07-25 10:48:23,477 [[api-my-app-v1-20.0.0].apiHttpListenerConfig.worker.58] INFO  esb.api-my-app-v1.get-blah.http.response - transactionID=f61b8053-70d2-11e7-b274-3639cc5335d0 creationTime=2017-07-25T10:48:23.254+10:00 txnState=END timeTaken=11

So far I have written this grok...

%{TIMESTAMP_ISO8601:timestamp}\s+%{DATA:thread}\s+%{LOGLEVEL:loglevel}\s+%{JAVACLASS:category}\s+-\s+%{GREEDYDATA:msgbody}

It gives me back this....

{
  "timestamp": [
    [
      "2017-07-25 10:48:23,477"
    ]
  ],
  "YEAR": [
    [
      "2017"
    ]
  ],
  "MONTHNUM": [
    [
      "07"
    ]
  ],
  "MONTHDAY": [
    [
      "25"
    ]
  ],
  "HOUR": [
    [
      "10",
      null
    ]
  ],
  "MINUTE": [
    [
      "48",
      null
    ]
  ],
  "SECOND": [
    [
      "23,477"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "thread": [
    [
      "[[api-my-app-v1-20.0.0].apiHttpListenerConfig.worker.58]"
    ]
  ],
  "loglevel": [
    [
      "INFO"
    ]
  ],
  "category": [
    [
      "esb.api-my-app-v1.get-blah.http.response"
    ]
  ],
  "msgbody": [
    [
      "transactionID=f61b8053-70d2-11e7-b274-3639cc5335d0 creationTime=2017-07-25T10:48:23.254+10:00 txnState=END timeTaken=11"
    ]
  ]
}

This is almost what I want. How can I split the msgbody from my current result into key value pairs?

thanks

Perhaps you could explain what you want to extract? Also if you want to extract key value pair, you can use the [kv filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html), once you have a string containing only key-value pairs. — baudsp, Jul 25 '17 at 08:12
Hi, Sorry for the vagueness. I'm re-worded my question to be more specific. Your note about kv filter seems like it could be useful to me. I'm going to ahve a look at it now. — Richie, Jul 25 '17 at 08:23

score 5 · Accepted Answer · answered Jul 25 '17 at 08:47

5

With the kv filter:

kv { 
   source => "msgbody" 
}

you'll have the key-pair values from the msgbody in fields in your result. Also you won't have to change your grok pattern if the keys change.

answered Jul 25 '17 at 08:47

baudsp

4,076
1
17
35

This looks elegant! But what if the "msgbody" had some non-keyvalue data? Will the kv filter ignore it? Like for example "transactionID=f61b8053-70d2-11e7-b274-3639cc5335d0 creationTime=2017-07-25T10:48:23.254+10:00 txnState=END timeTaken=11 - Successfully Executed the Request" – Mahesh H Viraktamath Mar 28 '19 at 12:46
@MaheshHViraktamath I'd say it depends on the params, it might get dropped without issue. It's simpler to keep only the KV data. In your example you could remove everything after the `-`, possibly with [mutate#gsub](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-gsub). – baudsp Mar 28 '19 at 14:39
@baudsp Ok, thank you. I will try to read the part after "-" in another field may be. – Mahesh H Viraktamath Mar 29 '19 at 06:12

writing grok pattern to split key value pairs

1 Answers1