3

I'm working on Azure cloud solution. I'm using IoT Hub connected to Kafka in order to process data coming from various IoT devices. What I'm facing is that all the data coming from multiple devices are stored in the same topic . However, I want to process the data of each device connected to IoT Hub to a specific topic in Kafka (Each device has its own Kafka topic)

The Toketi "Kafka Connect Source Connector for Azure IoT Hub" provides the following config file (edge node) 

connector.class=com.microsoft.azure.iot.kafka.connect.source.IotHubSourceConnector
name=AzureIotHubConnector
tasks.max=1
Kafka.Topic=IotTopic
IotHub.EventHubCompatibleName=iothub-toketi
IotHub.EventHubCompatibleEndpoint=sb://iothub-001.servicebus.windows.net/
IotHub.AccessKeyName=service
IotHub.AccessKeyValue=4KsdfiB9J899a+N3iwerjKwzeqbZUj1K//KKj1ye9i3=
IotHub.ConsumerGroup=$Default
IotHub.Partitions=4
IotHub.StartTime=2016-11-28T00:00:00Z
IotHub.Offsets=
BatchSize=100
ReceiveTimeout=60

It workes for one topic to store all data from multiple devices but i expect to do isolation between data coming from devices

Any solutions or ideas !!

Thanks

Anis Tissaoui
  • 834
  • 1
  • 7
  • 26
ben kheder rami
  • 31
  • 1
  • 5

1 Answers1

2

One of the solution is to use SMT (Single Message Transformation).

Source Connector flow contains several steps:

  • Poll data from external source as List<SourceRecord>
  • Transform each message (SourceRecord) using defined SMT (can be skipped if no Transformation is defined
  • Convert key and value of SourceRecord to Arrays of bytes.
  • Send Message via KafkaProducer to Kafka

Kafka Connect determine to which topic send a Message based on SourceRecord::topic field. Using SMT you can set proper topic value.

Pure Apache Kafka Connect doesn't have such Transformation. If you are using Confluent Platform some additional Transformation are available. To extract topic name you can use ExtractTopic. It has property, that is called field

More about whole concept of SMT can be found at Apache Kafka web page or Confluent web page

Bartosz Wardziński
  • 6,185
  • 1
  • 19
  • 30
  • i am not working in confluent plateform to use it's options(kafka on Azure) and SMT implentation in Kafka does not respond to my need. Any other solution could be found ?? – ben kheder rami Apr 18 '19 at 08:43
  • @benkhederrami, I think you can download connect-transformations from Confluent and install it in your Kafka Connect (https://www.confluent.io/connector/connect-transformations/). It can be done same way like standard connector. Connectors, Converters and Transformation can be installed as a plugin to Kafka Connect – Bartosz Wardziński Apr 18 '19 at 12:00
  • How could i extract a name from my message and use it as the topic name then store all messages that contains same name . I gave him a json message but he mensionned that he can stract field juste from struct message. – ben kheder rami Apr 25 '19 at 09:29