1

I have a MQTT broker and a Kafka broker running, I have used the kafka-connector: https://github.com/Landoop/stream-reactor, with the next configuration:

name=Mqtt-Source
connector.class=com.datamountaineer.streamreactor.connect.mqtt.source.MqttSourceConnector
tasks.max=1
connect.mqtt.kcql=INSERT INTO test SELECT * FROM + WITHCONVERTER=`com.datamountaineer.streamreactor.connect.converters.source.JsonSimpleConverter` WITHKEY(id)
connect.mqtt.connection.clean=true
connect.mqtt.connection.timeout=1000
connect.mqtt.connection.keep.alive=1000
connect.mqtt.client.id=test_mqtt_connector
connect.mqtt.converter.throw.on.error=true
connect.mqtt.hosts=tcp://mqtt-broker:1883
connect.mqtt.service.quality=1

In the kcql, I'm definning the field of the message that kafka should take as key, is it anyway to use the mqtt-topic as key? So I don't need to define the WITHKEY() in the kcql.

Asier Gomez
  • 6,034
  • 18
  • 52
  • 105

1 Answers1

0

I don't know about Landoop's KCQL, but assuming the topic is part of the message value, you can move it to the key like so

transforms=ReplaceKey,ExtractKey
transforms.ReplaceKey.type=org.apache.kafka.connect.transforms.ValueToKey
# change the field accordingly
transforms.ReplaceKey.fields=mqtt_topic
transforms.ExtractKey.type=org.apache.kafka.connect.transforms.ExtractField$Key
# make sure this is the same field as above
transforms.ExtractKey.field=mqtt_topic

If not, then you can staticly insert it

transforms=AddKey
transforms.AddKey.type=org.apache.kafka.connect.transforms.InsertField$Key
# The exclamation makes this a required field
transforms.AddKey.static.field=mqtt_topic!
transforms.AddKey.static.value="<<your topic name>>"

However, the above might not work with SELECT * FROM +, where you are selecting from all MQTT topics

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks, but the problem is that the KCQL need to define the key, how can I send the key from there? – Asier Gomez Nov 20 '18 at 08:25
  • The first solution you propouse, is throwing:`java.lang.NullPointerException at org.apache.kafka.connect.transforms.ValueToKey.applyWithSchema(ValueToKey.java:85)` if I put that paraghraph behind kcql command, any idea? – Asier Gomez Nov 20 '18 at 08:45
  • You're not supposed to literally copy it... As I commented in it, change accordingly... I cannot see your Kafka messages to know what that should be... And looking at the KCQL documentation or MQTT Landoop page, no you actually don't need a key, since keys can be null in Kafka messages anyway – OneCricketeer Nov 20 '18 at 11:16
  • If I don't add a key it fails asking for a key, I know in the documentation it doesn't need, but then it fails if you don't give a key – Asier Gomez Nov 20 '18 at 11:19