spring kafka thorws InstanceAlreadyExistsException exception after setting concurrency > 1

Question

I'm using spring-kafka, everything is working if I don't set the concurrency of ConcurrentKafkaListenerContainerFactory, when I set it to a number greater than 1, I get an exception:

javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=client-3

My config:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> 
    kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new 
    ConcurrentKafkaListenerContainerFactory<String, String>();

    factory.setConcurrency(kafkaConfig.getConcurrency());

    factory.getContainerProperties().setAckMode(AckMode.MANUAL);
    factory.setConsumerFactory(consumerFactory());
    return factory;
}

properties:

kafka.enable-auto-commit=false

kafka.client-id=client-1

kafka.concurrency=2

score 7 · Accepted Answer · edited Nov 23 '19 at 08:47

7

I have opened an issue for this on github. Setting a different client.id for each thread is not currently supported.

As a work-around, you could launch a separate KafkaMessageListenerContainer for each (which is what the ConcurrentMessageListenerContainer does internally).

EDIT

Although not ideal, you can omit the client.id and the kafka client will generate one for each (consumer-1, consumer-2 etc)

edited Nov 23 '19 at 08:47

Kalle Richter

8,008
26
77
177

answered Apr 25 '17 at 12:00

Gary Russell

166,535
14
146
179

1

I reproduced it; BTW, omitting the `client.id` works, if that's an option for you. – Gary Russell Apr 25 '17 at 12:17
1

The client.id is now unique for each consumer in the container by appending `-` (since 1.0.6). – Gary Russell May 10 '19 at 16:24

spring kafka thorws InstanceAlreadyExistsException exception after setting concurrency > 1

1 Answers1