Static membership in Kafka Connect

Question

I'm trying to implement the static membership in Kafka Connect. Our Kafka Connect cluster is deployed on K8S using the Strimzi Kafka operator.

I have tried putting the following config for the workers (in the KafkaConnect yaml):

connector.client.config.override.policy: All
consumer.group.instance.id: somethingsomething

And in the HttpSinkConnector class I have this:

@Override
public List<Map<String, String>> taskConfigs(int maxTasks) {
    List<Map<String, String>> configs = new ArrayList<>(maxTasks);
    for (int i = 0; i < maxTasks; i++) {
        Map<String, String> configCopy = new HashMap<>(this.configProps);
        configCopy.put("consumer.override.group.instance.id", Thread.currentThread().getName());

        configs.add(configCopy);
    }
    return configs;
}

This gave org.apache.kafka.common.errors.FencedInstanceIdException with some log - ...07:07:32,631 ERROR [Consumer instanceId=somethingsomething... because all tasks got somethingsomething as their group.instance.id although it should have get Thread.currentThread().getName().

I have also tried just the following (without the the worker configuration):

@Override
public List<Map<String, String>> taskConfigs(int maxTasks) {
    List<Map<String, String>> configs = new ArrayList<>(maxTasks);
    for (int i = 0; i < maxTasks; i++) {
        Map<String, String> configCopy = new HashMap<>(this.configProps);
        configCopy.put("consumer.group.instance.id", "somethingsomething");

        configs.add(configCopy);
    }
    return configs;
}

And that did nothing (no errors, no instanceId in logs) which mean I put this config value in the wrong place.

So how can I achieve static membership on Kafka Connect?

score 1 · Answer 1 · answered Jun 10 '21 at 10:42

1

Use consumer.override.group.instance.id instead of consumer.group.instance.id

answered Jun 10 '21 at 10:42

Sumeet

23
2

This was the first attempt as it is described in the question... – omer bar lev Jun 10 '21 at 13:34
I believe you are referring to connector.client.config.override.policy: All , i assume that you must have configured in kafka connect configuration. that is correct. I am referring to using consumer.override.group.instance.id in your connector config. instead of this => consumer.group.instance.id: somethingsomething use => consumer.override.group.instance.id: somethingsomething – Sumeet Jun 16 '21 at 04:11
Won't configure the connector the way you suggest will make all the tasks to be initialized with the same consumer.group.instance.id? I need each task(==consumer) to have a unique consumer.group.instance.id – omer bar lev Jun 16 '21 at 19:16

Static membership in Kafka Connect

1 Answers1