How can I aggregate terms based on a string prefix in Kibana4?

Question

One field which is flowing into our Kibana instance contains the logger name of Python applications. Those names represent the application structure, separated by dots. For example, we could have the following loggers:

a
a.b
a.b.c
a.d
b.c

I would like to set up a 2-Level pie chart in Kibana4 using the top-level logger (a and b) in this case as inner slices, and the second level loggers (a.b, a.d, b.c) as outer "donut".

"Unrolling" the pie-chart ans ASCIIfying the intended result would look like this:

==============++++   <-- representing the main aggregation (inner slices)
----oooooooooo****   <-- representing the sub-aggregation (outer donut)

With:

    =: a
    +: b
    -: a.b
    o: a.d
    *: b.c

In Kibana3, the logger names were actually split into separate tokens, so I would have tokens a, b, c and d available. Kibana4 does not seem to split them.

I tried to append a custom index mapping setting "index" to "analyzed", but that still left the field "unsplit". After additional thought, that would not really help me to get the output I want.

Instead, it might be helpful to take the incoming message, and store two additional fields in the index, representing the top-level and second-level logger names. So, an incoming value of "a.b.c" would be stored as "a" (the root logger), "a.b" (the first-level-logger) and "a.b.c" (the unmodified value) in the index.

That way I could set up the pie chart using the root-level logger as main aggregation, and the first-level logger as sub-aggregation.

Is this somehow possible?

In very short: I have a term like `foo.bar.baz.qux`. I would like to index that as `foo`, `foo.bar` and `foo.bar.baz.qux`. — exhuma, May 22 '15 at 19:15

score 0 · Answer 1 · answered May 23 '15 at 16:21

As long as you have a fixed number of levels you could simply define 3 patterns like.

FirstLevel: %{DATA:firstLevel}. FirstAndSecondLevel: %{GREEDYDATA:firstAndSecondLevel}. FullPath: %{GREEDYDATA:fullPath}

You could then use he grok filter to apply all three patterns on your message. You can either do 3 separate grok filters each with one of the patterns or you can put all three patterns in one grok filter and set break_on_match to false.

You should then be able to group for these fields in kibana.

How can I aggregate terms based on a string prefix in Kibana4?

1 Answers1