Kibana: Search within text for string

Question

I have A log message in Kibana that contains this:

org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:597)

Actual search that isn't returning results: log_message: "hibernate3"

If I search for "hibernate3" this message will not appear. I am using an Elasticsearch template and have indexed the field, but also want to be able to do case-insensitive full-text searching. Is this possible?

Template that is in use:

{
"template": "filebeat-*",
"mappings": {
    "mainProgram": {
        "properties": {
            "@timestamp": {
                "type": "date",
                "format": "strict_date_optional_time||epoch_millis"
            },
            "@version": {
                "type": "text"
            },
            "beat": {
                "properties": {
                    "hostname": {
                        "type": "text"
                    },
                    "name": {
                        "type": "text"
                    }
                }
            },
            "class_method": {
                "type": "text",
                "fielddata": "true",
                "index": "true"
            },
            "class_name": {
                "type": "text",
                "fielddata": "true"
            },
            "clientip": {
                "type": "ip",
                "index": "not_analyzed"
            },
            "count": {
                "type": "long"
            },
            "host": {
                "type": "text",
                "index": "not_analyzed"
            },
            "input_type": {
                "type": "text",
                "index": "not_analyzed"
            },
            "log_level": {
                "type": "text",
                "fielddata": "true",
                "index": "true"
            },
            "log_message": {
                "type": "text",
                "index": "true"
            },
            "log_timestamp": {
                "type": "text"
            },
            "log_ts": {
                "type": "long",
                "index": "not_analyzed"
            },
            "message": {
                "type": "text"
            },
            "offset": {
                "type": "long",
                "index": "not_analyzed"
            },
            "query_params": {
                "type": "text",
                "index": "true"
            },
            "sessionid": {
                "type": "text",
                "index": "true"
            },
            "source": {
                "type": "text",
                "index": "not_analyzed"
            },
            "tags": {
                "type": "text"
            },
            "thread": {
                "type": "text",
                "index": "true"
            },
            "type": {
                "type": "text"
            },
            "user_account_combo": {
                "type": "text",
                "index": "true"
            },
            "version": {
                "type": "text"
            }
        }
    },
    "access": {
        "properties": {
            "@timestamp": {
                "type": "date",
                "format": "strict_date_optional_time||epoch_millis"
            },
            "@version": {
                "type": "text"
            },
            "beat": {
                "properties": {
                    "hostname": {
                        "type": "text"
                    },
                    "name": {
                        "type": "text"
                    }
                }
            },
            "clientip": {
                "type": "ip",
                "index": "not_analyzed"
            },
            "count": {
                "type": "long",
                "index": "not_analyzed"
            },
            "host": {
                "type": "text",
                "index": "true"
            },
            "input_type": {
                "type": "text",
                "index": "not_analyzed"
            },
            "log_timestamp": {
                "type": "text"
            },
            "log_ts": {
                "type": "long",
                "index": "not_analyzed"
            },
            "message": {
                "type": "text"
            },
            "offset": {
                "type": "long",
                "index": "not_analyzed"
            },
            "query_params": {
                "type": "text",
                "index": "true"
            },
            "response_time": {
                "type": "long"
            },
            "sessionid": {
                "type": "text",
                "index": "true"
            },
            "source": {
                "type": "text",
                "index": "not_analyzed"
            },
            "statuscode": {
                "type": "long"
            },
            "tags": {
                "type": "text"
            },
            "thread": {
                "type": "text",
                "index": "true"
            },
            "type": {
                "type": "text",
                "index": "true"
            },
            "uripath": {
                "type": "text",
                "index": "true"
            },
            "user_account_combo": {
                "type": "text",
                "index": "true"
            },
            "verb": {
                "type": "text",
                "index": "true"
            }
        }
    }
}
}

Eljah · Answer 1 · 2020-10-14T06:51:55.143

22

message: *.hibernate3.*

also works (please note, that no quotes are needed for that)

edited Oct 14 '20 at 06:51

answered Oct 13 '20 at 21:07

Eljah

4,188
4
41
85

14

What if your query has spaces? i.e. `message:*the quick brown fox*` does not work, neither does enclosing it in double-quotes. Basic search features like this are half a century old, strange how reinventing this wheel today results in an inferior wheel. – Coder Guy Mar 30 '21 at 16:36
11

What @JonathanNeufeld said. It blows my mind that Kibana lacks some of these drop-dead basic textual query features. – Mass Dot Net Jun 02 '21 at 18:32
you don't need the asterisks – akostadinov Oct 18 '22 at 10:43
1

If your query has spaces you can use `%` (percent sign) like this: `message: "%Cannot open connection%"`. Based on @Olivier Tonglet answer. – luke Feb 20 '23 at 19:28

score 13 · Accepted Answer · edited Nov 09 '20 at 18:03

13

According to your scenario, what you're looking for is an analyzed type string which would first analyze the string and then index it. A quote from the doc.

In other words, index this field as full text.

Thus make sure that, you have your mapping of the necessary fields properly so that you'll be able to do a full-text search on the docs.

Assuming that, in Kibana if the log line is under the field message, you could simply search for the word by:

message:"hibernate3"

You might also want to refer this, to identify the variance between Term Based and Full-Text.

EDIT

Have the mapping of the field log_message as such:

"log_message": {
       "type": "string", <- to make it analyzed
       "index": "true"
}

Also try doing a wildcard search as such:

{"wildcard":{"log_message":"*.hibernate3.*"}}

edited Nov 09 '20 at 18:03

Willi Mentzel

27,862
20
113
121

answered Feb 28 '17 at 17:39

Kulasangar

9,046
5
51
82

For some reason this isn't working: log_message:".hibernate3." returns no results. -- Where log_message is a subset of message. My ElasticSearch template has this as indexed: log_message type text index TRUE – A_Elric Feb 28 '17 at 18:00
Updated template above – A_Elric Feb 28 '17 at 18:22
So you mean, you don't see a separate field called `log_message` in Kibana? – Kulasangar Mar 01 '17 at 04:07
And also could you try searching it as a `wildcard`, like `log_message:"*.hibernate3.*"` between asterisks? I've updated the answer! – Kulasangar Mar 01 '17 at 04:22
1

I should note that you are out-of-luck if your search query contains spaces. Neither `log_message:"*the quick brown*"` nor `log_message:*the quick brown*` work as intended. Why is KQL so non-intuitive like this? – Coder Guy Mar 30 '21 at 16:38
2

@JonathanNeufeld Simply enclose the search with double-quotes, e.g. ```message:"the quick brown fox"```. You don't need asterisks. – RCross Nov 18 '21 at 11:56
An ugly hack that seems to work is to replace all spaces with asterisks. While not the same as a properly formatted search, it can help filter down to fewer log messages. `log_message: *the*quick*brown*` ¯\\_(ツ)_/¯ – bsyk Aug 15 '23 at 16:53

score 10 · Answer 3 · answered Nov 03 '21 at 16:22

10

With Kibana 6.4.1 I used the "%" as wildcard.

message: %hibernate3%

answered Nov 03 '21 at 16:22

Olivier Tonglet

3,312
24
40

score 4 · Answer 4 · answered Mar 25 '22 at 14:21

4

For me it was because I was using the ".keyword".

My key was called "message" and I had "message" and "message.keyword" available.

Full text search isn't working on ".keyword".

Not working :

message.keyword : hello

Working :

message : hello

answered Mar 25 '22 at 14:21

Doctor

7,115
4
37
55

In Kibana or OpenSearch dashboard, one has to `add filter` -> `message` `is` `whole word or multiple quoted words`. It will not catch partial words it seems. – akostadinov Oct 18 '22 at 10:47

Kibana: Search within text for string

4 Answers4

Linked