1

I'm using elasticsearch sink kafka-connector to index messages from multiple kafka topics to elasticsearch. I have my topics with camelCase naming, and I can't change it. So when starting up the ES sink connector, it does not index anything because elaticsearch has problems with non-lowercase index names.

I know I can use topic.index.map property to manually convert topic name to index.
topic.index.map=myTopic1:mytopic1, myTopic2:mytopic2,...

Is there a way to convert to lowercase automatically? I have dozens of topics to convert, and I suspect it to be around hundred soon.

Viacheslav Shalamov
  • 4,149
  • 6
  • 44
  • 66
  • Not sure how to automatise his, but you can have upper letter in alias, so maybe you could create a mapping between alias (with upper) and generated lowercase index ? – LeBigCat Feb 14 '19 at 15:00

2 Answers2

1

Found out that since 5.1 they do that automatically, if mapping is not specified for the topic.
from here:

final String indexOverride = topicToIndexMap.get(topic);
String index = indexOverride != null ? indexOverride : topic.toLowerCase();

See this commit for details.

Viacheslav Shalamov
  • 4,149
  • 6
  • 44
  • 66
1

As of recent versions of the Elasticsearch sink connector this is done automatically. The PR that fixed this was https://github.com/confluentinc/kafka-connect-elasticsearch/pull/251

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92