1

Using sumologic, how can I perform queries on log entries where the message is partially json (so not entirely)?

Example entry:

[2020-04-30 02:10:32] production.WARNING: We were rate limited {"class":"App\WebhookService\WebhookExecutor","headers":{"Date":["Thu, 30 Apr 2020 02:10:32 GMT"],"Content-Type":["application/json"],"Content-Length":["88"],"Connection":["keep-alive"],"Set-Cookie":...

I have a lot of these kinds of errors with different class paths, some are webhooks executions, others are different objects. My end goal is to be able to count the log entries for each class name, like this:

WebhookExecutor     | 156
SyncDataService     | 48
DeleteService       | 13

I'm starting with something like:

"We were rate limited" | json auto | json "class"

My expectation with ^ was that it would create a column called class in my Log Search results that I could then group by, count, etc.

Ben
  • 60,438
  • 111
  • 314
  • 488

1 Answers1

1

Not sure what do you mean by "partially Json"? I.e.:

  1. that the log message contains other text and a proper Json
  2. or that the message contains something which is similar to Json, but not really a Json - e.g. a prefix of a Json message

Assuming Option 1 (other text + proper Json)

You need to extract your Json to a proper field of its own. Otherwise | json auto is not able to pick it up.

So:

"We were rate limited"
| parse "We were rate limited *" as limitedJson
| json auto
| count by class

should do the job. The first line is not even needed, because parse does the filtering too.

Assuming Option 2 (no proper Json, just something similar)

It's harder in such a case, but still doable.

I haven't tested it, but something like this should work:

"We were rate limited"
| parse "\"class\":\"*\"" as class
| count by class

Here escaping the nested quotes doesn't look well.

Disclaimer: I am currently employed by Sumo Logic.

Community
  • 1
  • 1
Grzegorz Oledzki
  • 23,614
  • 16
  • 68
  • 106
  • I appreciate the follow up! "partially json" I'm referring to the example log entry I included above, where it begins with a message, then the json begins. I don't quite understand _why_ parse is needed since `json auto`'s documentation seems to suggest it's intelligently able to extract the json? https://help.sumologic.com/05Search/Search-Query-Language/01-Parse-Operators/03-Parse-JSON-Formatted-Logs#json-auto-option. Even so, Option 2 works perfectly for my use case, thank you! – Ben Apr 30 '20 at 11:33
  • Interesting. I didn't know that to be honest. Could it be "In cases where content appears at the end of the message after the json blob, the extraction could fail."? Is what you pasted a real log message or redacted one? i.e. is the ellipsis part of the actual log line or just for posting to Stackoverflow? – Grzegorz Oledzki Apr 30 '20 at 17:29
  • 1
    The `...` was redacted. But you're right. The log entry's context is being cut off because the json blob is too big, so that messed up the json. Good call :) – Ben May 08 '20 at 21:09