0

I am using a Mongo-Connector targeting elasticsearch. This works great for keeping elasticsearch up to date, but I have a problem with one of the fields because it contains an '_'. The data is being replicated/streamed from mongodb continually if I run a rename/reindex the new documents will start showing up with underscores again.

Kibana does not support underscores at the start of a field. What is the best practice for handling this?

I have filed an issue with elastic2-doc-manager for Mongo-Connector to support ingest nodes, but this feels like a much bigger issue with kibana all my attempts at fixing this issue using scripted fields and renaming the field have failed.

This seems like a huge problem. I see underscores in data everywhere, seems like a very poor decision on the side of the kibana team.

Kibana Error: enter image description here

I have found some github referencese to this issue, but no work arounds.

Aaron
  • 1,031
  • 10
  • 23
  • What you're calling a "poor decision on the side of the kibana team" is an old Java instance variable naming convention, so it would have made sense and been within the scope of best practice for the devs who wrote Elasticsearch in Java in 2010 to name reserved things in this way. It would additionally have made sense as best practice for Kibana to enforce it. In the past decade this convention has gone from progressively falling out of favor to being outright bad practice, but 10-15+ years ago it would've been taught by any conceptually/stylistically rigorous Computer Science degree program. – Allison Feb 25 '22 at 03:09

1 Answers1

2

Fields beginning with _ are reserved for use within Elasticsearch. Kibana does not support fields with _ currently, at least not yet. A request for this - https://github.com/elastic/kibana/issues/14856 is still open.

Until then if you would like to use the field in visualizations etc, I believe you need to rename it.

While you can't rename the field easily without using logstash or filebeat and Mongo-Connector doesn't support either of them you can instead use a scripted field as below to create a new filed and copy the _ field's value. That way you can use the new field to visualize etc. Add a new scripted field for ex. itemType with the below script and see if it works.

doc['_itemType.keyword'].value

Please note though that only keyword fields can be used like this, text type fields won't work. If your _itemType field is of type text, modify the mapping to include a sub field keyword of keyword type under _itemType and try the scripted field.

ben5556
  • 2,915
  • 2
  • 11
  • 16
  • My question is exactly that how do I rename this field? Documents are being added and updated in a live stream from mongo db. – Aaron Oct 26 '18 at 16:48
  • That did it I had tried scripted fields several times but did not include 'keyword' – Aaron Oct 29 '18 at 17:03