1

We are using Kafka-Mqtt connector from Confluent team. Is there any way to publish data from multiple Mqtt topics to multiple Kafka topics with same name as of Mqtt , i.e test1 in Mqtt -> test1 in kafka so on. ?

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
Kartik D
  • 11
  • 3

1 Answers1

2

You can run multiple MQTT Source connectors in order to replicate data from distinct topics:

For example, 

{
  "config" : {
    "name" : "MqttSourceConnector1",
    "connector.class" : "io.confluent.connect.mqtt.MqttSourceConnector",
    "tasks.max" : "1",
    "mqtt.server.uri" : "< Required Configuration >",
    "mqtt.topics" : "topic_1"
  }
}

and, 

  {
      "config" : {
        "name" : "MqttSourceConnector2",
        "connector.class" : "io.confluent.connect.mqtt.MqttSourceConnector",
        "tasks.max" : "1",
        "mqtt.server.uri" : "< Required Configuration >",
        "mqtt.topics" : "topic_2"
      }
    }
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
  • 1
    This applies to all Connects, not just MQTT – OneCricketeer Sep 10 '18 at 13:20
  • @cricket_007 Correct. – Giorgos Myrianthous Sep 10 '18 at 13:21
  • @GiorgosMyrianthous what if I have many MQTT topics ? Isn't it too many connectors ? What about performance ? – Yadu Krishnan Sep 10 '18 at 17:21
  • Number of connectors should not affect performance. Performance is affected by the volume of data which is replicated from/to kafka/mqtt. You need to assign a reasonable amount of resources but as I said number of connectors does not have a major impact on the performance of Kafka Connect. If you want to minimize that impact though, just increase polling intervals. – Giorgos Myrianthous Sep 10 '18 at 17:32
  • @GiorgosMyrianthous Thanks for the reply. One concern I have is that we have many mqtt topics, and some are dynamically created (targeted to a particular use etc, by adding userId to topic (eg: topic_userId )). In this case, it is cumbersome to add connector for each topic separately. Any other way u can suggest ? – Yadu Krishnan Sep 11 '18 at 04:57
  • How about writing a bash script that generates (and calls Kafka Connect REST API) for multiple topics? – Giorgos Myrianthous Sep 11 '18 at 05:42