Grok multiline recipe?

Question

I'm still pretty new to logstash parsing out a multiline log message still seems a bit intimidating but I'm hoping that for what I'm trying to do (aka, parse logstash logs) this is pretty tried and tested patterns and hoping someone might be able to point me to a working solution.

Basically I have an error like this:

[2014-10-15 22:34:36,958][DEBUG][action.search.type       ] [kafka] [logstash-2014.10.15][1], node[RBRz8xNgQQKsAnEgbjqVTw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4511ca8d] lastShard [true]
org.elasticsearch.search.SearchParseException: [logstash-2014.10.15][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"facets":{"stats":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:200"}},{"query_string":{"query":"status:304"}},{"query_string":{"query":"_missing_:status"}},{"query_string":{"query":"type:\"nginx-error\""}},{"query_string":{"query":"type:(-\"nginx-*\")"}},{"query_string":{"query":"status:[308 TO 999]"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577151,"to":1413412477151}}}]}}}}}}},"stats_Success":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:200"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577151,"to":1413412477151}}}]}}}}}}},"stats_Cache Hits":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:304"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577151,"to":1413412477151}}}]}}}}}}},"stats_No Status":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"_missing_:status"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}},"stats_NGINX error":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"type:\"nginx-error\""}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}},"stats_Not Nginx":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"type:(-\"nginx-*\")"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}},"stats_Failure":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:[308 TO 999]"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}}},"size":0}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:660)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:516)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:488)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet [stats]: field [host] isn't a number field, but a string
    at org.elasticsearch.search.facet.statistical.StatisticalFacetParser.parse(StatisticalFacetParser.java:132)
    at org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:644)
    ... 9 more

Which seems to be balking at my use of "host" in the Kibana dashboard as a numeric. I'm not actually that interested in the actual error and more interested in being able to parse this in an effective logstash sort of way so that I can look into Kibana and see a sensible error message.

I'm assuming that it would involve some use of the multiline { ... } codec for my file input but looking at that message it looks pretty hairy ... surely there is a proverbial "wheel" out there somewhere which I don't need to invent?

score 1 · Answer 1 · edited Oct 16 '14 at 08:02

1

multiline {...} is the way to go. Try the snippet below in file type of input.

codec => multiline
{
    pattern => "(^\d+\serror)|(^.+Exception: .+)|(^\s+at .+)|(^\?s+... \d+ more)|(^\s*Caused by:.+)|(^.+ \d+ common frames omitted)
    what => "previous"
    multiline_tag => "multi_tagged"
}

edited Oct 16 '14 at 08:02

ken

8,763
11
72
133

answered Oct 16 '14 at 00:05

user3195649

437
1
6
11

score 1 · Answer 2 · answered Oct 16 '14 at 02:48

1

Each exception starts with the date, so key off of that. Here's a 'filter' example, but it holds for codec:

filter {
    multiline {
      negate => 'true'
      pattern => "^\[%{YEAR}"
      what => 'previous'
    }
  }
}

answered Oct 16 '14 at 02:48

Alain Collins

16,268
2
32
55

1

The codec is the preferred way, since then the message is complete before the filter stanza starts. I imagine they'll deprecate the filter eventually. I posted the filter example because I had it handy, and the pattern seemed more clean than others. – Alain Collins Oct 16 '14 at 16:10

Grok multiline recipe?

2 Answers2