Kibana Logstash ElasticSearch | Unindexed fields cannot be searched

Question

I am exploring ELK stack and coming across an issue.

I have generated logs, forwarded the logs to logstash, logs are in JSON format so they are pushed directly into ES with only JSON filter in Logstash config, connected and started Kibana pointing to the ES.

Logstash Config:

 filter {
  json {
    source => "message"
  }

Now I have indexes created for each day's log and Kibana happily shows all of the logs from all indexes.

My issue is: there are many fields in logs which are not enabled/indexed for filtering in Kibana. When I try to add them to the filer in Kibana, it says "unindexed fields cannot be searched".

Note: these are not sys/apache log. There are custom logs in JSON format.

Log format:

{"message":"ResponseDetails","@version":"1","@timestamp":"2015-05-23T03:18:51.782Z","type":"myGateway","file":"/tmp/myGatewayy.logstash","host":"localhost","offset":"1072","data":"text/javascript","statusCode":200,"correlationId":"a017db4ebf411edd3a79c6f86a3c0c2f","docType":"myGateway","level":"info","timestamp":"2015-05-23T03:15:58.796Z"}

fields like 'statusCode', 'correlationId' are not getting indexed. Any reason why?

Do I need to give a Mapping file to ES to ask it to index either all or given fields?

You should try adding an [index template](http://www.pipebug.com/elasticsearch-logstash-kibana-4-mapping-4.html) with custom fields definitions. — Val, May 27 '15 at 03:04
I tried it. I still see my fields unindexed. When I go to Visualize and try to choose Term Filter for X/Y axis, not all fields are showing up in there. This is getting bit frustrating for me to get it working now, as I don't see much documentation/reference about it. — rohit12sh, Jun 02 '15 at 16:55

score 48 · Accepted Answer · edited Dec 18 '17 at 18:42

48

You've updated the Kibana field list?

Kibana.
Settings.
Reload field list.

Newer version:

Kibana.
Management.
Refresh icon on the top right.

edited Dec 18 '17 at 18:42

Sniipe

1,116
3
13
28

answered Jul 02 '15 at 07:47

Busindre

604
6
3

Worked for me too. Thanks! – Psychic Rush Nov 12 '18 at 13:25
2

Should be a new step inserted before 3 under `Newer Version`: "Index Patterns" – Indigenuity Mar 09 '20 at 16:08
Still doesn't work for me.... – jtlz2 Jul 07 '22 at 07:21
If you are using Opensearch (currently 2.7): Navigation > `Stack Management` > `Index Patterns` > click the index pattern > click the `Refresh Field List` icon - the count of `number of fields` should increase. – Mr. C Aug 02 '23 at 12:38

score 8 · Answer 2 · answered Sep 18 '18 at 09:13

8

As of 6.4.0:

The warning description puts it very simply:

Management > Index Patterns > Select your Index > Press the refresh button in the top right corner.

answered Sep 18 '18 at 09:13

alex351

1,826
1
21
32

score 1 · Answer 3 · answered Jul 07 '23 at 14:10

1

On the newer version (2023), you need to update the Kibana field list from here:

Management
Stack Management
Index Patterns
Choose the pattern
Hit the button "Refresh field list"

answered Jul 07 '23 at 14:10

Dherik

17,757
11
115
164

score 0 · Answer 4 · answered Oct 30 '20 at 08:35

0

If you try to refresh and you can't solve it try to change index.blocks.write: "false"

enter image description here

answered Oct 30 '20 at 08:35

Musab Dogan

1,811
1
6
8

Kibana Logstash ElasticSearch | Unindexed fields cannot be searched

4 Answers4

Linked