1

Any option in Kafka Connect to specify from which partition specifically to read the messages. Basically, I am looking for an option in Kafka Connects to manually assign a list of partitions to read.

Similar to assign() method in KafkaConsumer API

https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#assign(java.util.Collection)

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Suraj
  • 97
  • 5

1 Answers1

0

You can't listen to only specific partition in Kafka Connect.

But you can achieve functionality of inserting messages from only specific partitions.

To have such feature you need to implement your custom Transformation. If Transformation returns null message is skipped, so your custom Transformation must return null for unwanted partitions.

Sample code will be as follow:

public class PartitionFilter <R extends ConnectRecord<R>> implements Transformation<R> {

    public static final ConfigDef CONFIG_DEF = new ConfigDef();

    @Override
    public void configure(Map<String, ?> props) {
        final SimpleConfig config = new SimpleConfig(CONFIG_DEF, props);
    }

    @Override
    public R apply(R record) {
        int neededPartition = 1; // some parititon
        if (record.kafkaPartition() != neededPartition)
           return null;
        return record;
    }

    @Override
    public void close() {
    }

    @Override
    public ConfigDef config() {
        return CONFIG_DEF;
    }
}

More information about transformations can be found: https://kafka.apache.org/documentation/#connect_transforms

Bartosz Wardziński
  • 6,185
  • 1
  • 19
  • 30
  • Is there a way to configure Kafka Connect to listen to only specific partitions? – Suraj Mar 30 '19 at 23:03
  • @Suraj, You can't listen to only specific partition (I've updated my answer), but you can achieve similar functionality by using STM (Single Message Transformation) - insert messages only from specific partition. – Bartosz Wardziński Mar 31 '19 at 11:50