1

I've been working with MQTT for a while, but I'm a newbie in Kafka. I'm dealing with a scenario in which N MQTT clients are publishing messages on M topics, where N and M can freely range from half a dozen to several hundred. They publish their messages on dynamically generated topics, therefore I don't know them a priori. I'm using Kafka Connect to ingest these messages into Kafka where they will undergo processing using KSQL. However, it looks like all of the incoming messages are getting stored in the same Kafka topic, the one that I can specify in the Connect configuration file. Therefore, even if I can subscribe to "#" in order to get all of the incoming MQTT messages into Kafka, they're all going to be stored in the same Kafka topic. I've been studying SMTs and specifically RegexRouter, but I can't figure out how to make it work. I also know someone suggests running multiple instances of Connect, and handle each topic separately, but what if the number nor the format of the topics are known a priori? I'd just need a 1:1 mapping between the original MQTT topic and the corresponding Kafka topic. Same name. Is there a way?

e.g. MQTT message published on "home/groundfloor/temp" goes into Kafka under "home/groundfloor/temp"

Input messages are very simple strings. Any help or hint would be appreciated! Thanks

DaleCooper
  • 11
  • 2
  • You can use multiple connector configurations in a single connect instance/cluster, but source tasks generally only have one input topic + Kafka topics cannot have slashes in the name... Can you share how you're trying to use RegexRouter? – OneCricketeer Feb 21 '21 at 18:33
  • Ouch, didn't know slashes are not allowed but that's not a big problem. The issue with multiple connectors is that I don't know how many there will be, and the number could be very huge. My try with RegexRouter was to route input topic filter '#' into a new topic having the original topic name, but seems not to be possible sadly. – DaleCooper Feb 23 '21 at 11:07

0 Answers0