How to consume Kafka Topic in Consumer MQ Topic

Question

I have a requirement where I need to consume Kafka Topic and write it into MQ Topic. Can someone advise me the best way to do it, I am new to Kafka.

I have read about the IBM MQ Connector in confluent but could not get the idea how to implement it.

score 1 · Answer 1 · answered May 28 '19 at 08:35

1

The best way to move data from Kafka to MQ is to use the IBM MQ sink connector: https://github.com/ibm-messaging/kafka-connect-mq-sink

This is a Kafka Connect connector. The README contains details for building and running it.

answered May 28 '19 at 08:35

Mickael Maison

25,067
7
71
68

Thank you @Mickael for quick response. I am following your suggestion and will try to implement the same. – Touseef Zaki May 28 '19 at 08:50

score 0 · Accepted Answer · answered May 28 '19 at 08:52

Kafka has a component called Kafka Connect. It is used to read and write data to/from Kafka into other systems such as Database in your case MQ.

Kafka connect can have two kind of connectors

Source connectors - Read data from an external system and write to Kafka (For eg. Read inserted/modified rows from a table in DB and insert into a topic in Kafka)

Sink Connector - Read message from Kafka write to external system.

The link you have added is a source connector, it will read messages from the MQ and write to Kafka.

For simple use case you do not need Kafka connect. You can write a simple Kafka consumer that will read data from Kafka topic and write it to MQ.

    Properties props = new Properties();
     props.put("bootstrap.servers", "localhost:9092");
     props.put("group.id", "test");
     props.put("enable.auto.commit", "true");
     props.put("auto.commit.interval.ms", "1000");
     props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
     props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
     KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
     consumer.subscribe(Arrays.asList("foo", "bar"));
     while (true) {
         ConsumerRecords<String, String> records = consumer.poll(100);
         for (ConsumerRecord<String, String> record : records)
//Insert code to append to MQ here
             System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
     }

Thanks @asolanki for response, just wanted to know if i have XML messages stored in Topic then do i need to parse it while writing into MQ Topic? — Touseef Zaki, May 28 '19 at 09:01
Depends on the schema of message you are storing in MQ. If it is xml then not required, whereas say if it is json or other format then you do need. — asolanki, May 28 '19 at 11:25

How to consume Kafka Topic in Consumer MQ Topic

2 Answers2