kibana 5 scripted field for regex match

Question

I want to add scritpted field in Kibana 5 to get stored proc name from message. To be able to visualize number of errors per each SP. I have field "message" where I can see error text:

    "[2017-02-03 05:04:51,087] @ MyApp.Common.Server.Logging.ExceptionLogger [ERROR]: XmlWebServices Exception
User:
  Name:    XXXXXXXXXXXXXXXXXXXXXXX 
  Email:   926715@test.com
  User ID: 926715 (PAID)

Web Server: PERFTESTSRV
Exception:
  Type:    MyApp.Common.Server.DatabaseException
  Message: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
  Source:  MyApp.Common.Server
   Database:  MyDB
   Cmd Type:  StoredProcedure
   Cmd Text:  spGetData
   Trans:     YES
   Trans Lvl: Unspecified"

Guide: https://www.elastic.co/blog/using-painless-kibana-scripted-fields

My plan is to add something like as a Painless script:

def m = /(?:Cmd\sText:\s*)[a-zA-Z]{1,}/.matcher(doc['message'].value);
if ( m.matches() ) {
   return m.group(1)
} else {
   return "no match"
}

And also I've tried

def tst = doc['message'].value;
if (tst != null)
{
def m = /(?:User\sID:\s*)[0-9]{1,}/.matcher(tst);
if ( m.matches() ) {
   return m.group(1)
}
} else {
   return "no match"
}

How I can address doc['message'].value? When I try to do so I got error "Courier Fetch: 5 of 5 shards failed." When I try doc['message.keyword'].value, I do not have full message inside. I do not understand where I can learn the structure of what message have inside and how I can refer it?

How does your mapping look like for `message`? And what're you getting for your scripted field when you only have this within your script: **doc['message.keyword'].value**? — Kulasangar, Feb 08 '17 at 08:42
Not sure how to get mapping for message Here is what I see in Kibana http://joxi.ru/5mdvBMjSkelGdA When I have doc['message.keyword'].value I have following results for the type of errors I need I have only " - " as a result — Kristina Kucherova, Feb 08 '17 at 11:25
Are you getting the whole message as expected? Or is there something missing? What do you mean by ` need I have only " - " as a result`? — Kulasangar, Feb 08 '17 at 11:29
For this type of errors I need I have doc['message.keyword'].value dash as a results, and in "message" there is somewhat looking like data in screenshot (or in my example in the question with error text) there are some error for which results for field doc['message.keyword'].value are the same as field message. But unfortunately not for those I need. — Kristina Kucherova, Feb 08 '17 at 11:38
for this type (ERROR) doc['message.keyword'].value is null - I can see it in JSON markup, but still there is message inside "message" tag — Kristina Kucherova, Feb 08 '17 at 12:01

score 1 · Answer 1 · answered Feb 08 '17 at 12:36

I assume that problem with lenght of message. It is too long to be type "keyword". It should be type "text" which is not supported by painless.

https://www.elastic.co/blog/using-painless-kibana-scripted-fields

Both Painless and Lucene expressions operate on fields stored in doc_values. So >for string data, you will need to have the string to be stored in data type >keyword. Scripted fields based on Painless also cannot operate directly on >_source.

https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html_italic_

A field to index structured content such as email addresses, hostnames, status >codes, zip codes or tags. If you need to index full text content such as email bodies or product >descriptions, it is likely that you should rather use a text field.

kibana 5 scripted field for regex match

1 Answers1