Elasticsearch field mapping is showing date for version field

Question

I have recently setup a Kafka sink-connector to an Elastic pipeline and I noticed error in conversion of one field version which is definitely a text value. I checked the input format schema and I see it is being sent as text and I did not find any conversion for mapping of field I would have put. Any suggestion if something I should check to see where the field mapping information is getting populated for only this field ? Also, to update the mapping of field for all indices (index-0001, index-0002) is there an option to update the mapping for alias name ?

score 1 · Accepted Answer · answered Jun 04 '21 at 19:07

1

No, you can't set a mapping for a alias. It seems to me, that you never defined a mapping by yourself but relying on the dybamic mapping feature of elasticseaech. This feature will analyse the first document carrying a new field and the field value. Based on that analysis elastic will try an educated guess and set a mapping for that field in this mapping of the given index.

While this feature is very handy it's sometimes not what you expect. The best practice here is to ingest few docs and let elastic do the work. After taht we're taking the auto generated mapping and manually tweaking it until it's perfect. This mapping we're storing in a index template which will be auto-appiled to the new index when it's name matches the templates pattern.

In your case, look at

GET index-1/_mapping

and copy it into a editor in order to fix/complete the mapping.

Then add the bespoke mapping to a index template like that:

PUT _index_template/template_1
{
  "index_patterns": ["index-*"],
  "template": {
    "mappings": {
      "_source": {
        "enabled": true
      },
      "properties": {
        "version": {
          "type": "keyword"
        },
       ...more fields...

The very next index will have a priper mapping and you will be also able to reindex (and delete afterwards) the wrong mapped indicies into new ones using

POST _reindex
{
  "source" : {"index" : "index-1"},
  "dest" : { "index" : "index-1_new"}
}

Here are the related docs:

answered Jun 04 '21 at 19:07

ibexit

3,465
1
11
25

Thanks very much @ibexit for nicely explaining it I do see that I have not defined any mapping at the consumer side but there was an avro schema definition at consumer side which we are passing in the value converter field as well in the kafka connector `"value.converter": "io.confluent.connect.avro.AvroConverter"` Does elastic connector not learn from this ? – Learner Jun 04 '21 at 20:37
Sorry, but I´m not a kafka expert at all. I did a small research on this topic and found serveral approaches. One explaniation could be a wrong7incomplete avro schema. Ahve you set "value.converter.schema.registry.url" too? One of the best resources I found is here: https://teke42.wordpress.com/2020/04/21/how-do-i-stream-data-from-kafka-to-elasticsearch/ If you have any questions about elastic, just drop me a line. Good luck and feel free to share your solution when you figured it out. – ibexit Jun 04 '21 at 23:23
Thanks @ibexit very much for adding details Yeah, I have added schema registry url too and Avro schema looks good as the field type defined is string `type": ["null", "string"]` I guess its more of elastic server's own learning from data – Learner Jun 05 '21 at 02:03
I tried adding the mapping to template but I got "duplicate field" error map","caused_by":{"type":"json_parse_exception","reason":"Duplicate field ''\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 14, column: 7]"}},"status":400} I had added below section in the template , I checked there is no existing mapping in the template "mappings": { "properties": { "version": { "type": "keyword" } } } – Learner Jun 07 '21 at 16:40
Looks like the incoming (from perspective of elastic) json document has duplicate fields. Please verify the integrity ouf your output format. – ibexit Jun 07 '21 at 17:36
surprisingly, I restarted the process and I was able to submit the job .. happy and sad now :D Thanks very much for answering queries – Learner Jun 07 '21 at 19:17
Great job! Would you mind to share your answer? – ibexit Jun 07 '21 at 19:38

score 0 · Answer 2 · answered Jun 07 '21 at 22:52

0

I used the mappings suggest from @ibbexit and after that restarted the kafka connect process and it worked

answered Jun 07 '21 at 22:52

Learner

1,544
8
29
55

Elasticsearch field mapping is showing date for version field

2 Answers2