Apache Beam KafkaIO producer routing different messages to different topics

Question

I have a usecase where the incoming data has a key that identifies different type of the data. There's a single input kafka topic where all types of data are thrown at it. The beam pipeline reads all the messages from the input kafka topic and has to route to different kafka topics depending on the key.

As present, KafkaIO doesn't support writing to multiple topics using a single producer. The following code is inner working code of the KafkaIO.write()

final class AutoValue_KafkaIO_Write<K, V> extends Write<K, V> {
    private final String topic;
    private final WriteRecords<K, V> writeRecordsTransform;

    private AutoValue_KafkaIO_Write(@Nullable String topic, WriteRecords<K, V> writeRecordsTransform) {
        this.topic = topic;
        this.writeRecordsTransform = writeRecordsTransform;
    }

How to do it using kafkaIO producer of apache beam?

score 0 · Accepted Answer · answered Jul 05 '20 at 19:53

After going through days trying to achieve message routing, at present, kafkaIO doesn't support message routing to different topics.

A work around is to create a kafka producer for every different topic and segregate the elements to different pcollections depending on which send to different kafka topics.

Apache Beam KafkaIO producer routing different messages to different topics

1 Answers1