0

I have a EFK setup to centrally store and query logs of all the application services. There are some services that process events from messages.

I want to query how many unique events (transaction-number: UUID) has failed per service, considering there will be duplicate error message as the message-broker is configured to redeliver messages 5 times upon any runtime exceptions. I filtered the services with its name, but with Painless script I couldn't substring the transaction-number from the message because of this error, Fielddata is disabled on text fields...Alternatively use a keyword field. Is it advisable to set keyword for a log-message field? How to query such things in Kibana?

vvra
  • 2,832
  • 5
  • 38
  • 82

1 Answers1

0

Fielddata=true is needed to enable sorting, aggregations and scripting on a text field. The drawback is that it uses a lot of heap space and will lead to expensive operations.

Your query to find the unique events per service sounds like an aggregation to me. You want to know the number of "documents" (rows/items) per service, per transaction UUID which you substring from the error message, right? I would suggest you make a new field of type keyword for the transaction UUID. That way you do the substring at index time which will give you better performance. Then you can make an aggregation on service with sub-aggregation on your newly created transaction UUID field and that will give you the counts.

Of course this only makes sense if you'll have regular use for this field, but I imagine you will.

SylarBenes
  • 411
  • 3
  • 7