KafkaMessageListenerContainer vs ConcurrentMessageListenerContainer

Question

If I use listener.concurrency property, would this use KafkaMessageListenerContainer or ConcurrentMessageListenerContainer? Note I have not defined any beans explicitly and I leave the required bean creation to spring boot.

I wanted to know if my understanding is correct. I would want parallelism in consuming the message from topic which has got multiple partitions say 3. So is it fine if I configure it in application.yml file as follows. I have tried below configuration and see 3 consumers were created.

spring:
   kafka:
     consumer:
        group-id: group-id
        key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
        value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
     producer:
        key-serializer: org.apache.kafka.common.serialization.StringSerializer
        value-serializer: org.apache.kafka.common.serialization.StringSerializer
     listener:
        concurrency: 3

So my question is, do I have to create a bean to configure ConcurrentMessageListenerContainer or the above listener configuration would internally would use ConcurrentMessageListenerContainer instead of KafkaMessageListenerContainer when it sees listener.concurrency=3?

Rohit Yadav · Accepted Answer · 2019-10-27T04:10:59.427

spring.kafka.listener.concurrency= # Number of threads to run in the listener containers.

Concurrency is the property of ConcurrentMessageListenerContainer. If we are defining this property in application.yml means we want to use ConcurrentMessageListenerContainer.

In Kafka, only one consumer is allowed to read data from one partition at a time. Using KafkaMessageListenerContainer will provide only single threaded message listener consumer. KafkaMessageListenerContainer

Using ConcurrentMessageListenerContainer will provide the number of KafkaMessageListenerContainer defined in the Concurrency property in application.yml.

We can get the same behavior using following Listener using Java code.

@Configuration
@EnableKafka
public class KafkaListenerConfiguration {

    @Autowired
    private Environment environment;

    @Bean
    public KafkaListenerContainerFactory<?> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.setConcurrency(3);
        factory.setBatchListener(true);
        return factory;
    }

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, environment.getProperty("spring.kafka.bootstrap-servers"));
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "1111");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        return props;
    }
}

I would suggest you to go through the below link to have a practical example of the same.

https://howtoprogram.xyz/2016/09/25/spring-kafka-multi-threaded-message-consumption/

Does that mean -> if "concurrency: 3" is set, one consumer (pod) will pull 3 messages at a time and process it parallelly? I Am a bit confused, can you help? — Shalaka Deshpande, Feb 11 '22 at 06:09
think of the concurrent consumers of a consumer group as totally independent consumers each consuming data from a partition. If there are 3 partitions of the topic and 3 concurrent consumers, each concurrent consumer would be able to process data parallely. More explanation here https://stackoverflow.com/questions/54727660/spring-kafka-messagelistenercontainer — Mrinal Bhattacharjee, Jun 22 '22 at 18:34

KafkaMessageListenerContainer vs ConcurrentMessageListenerContainer

1 Answers1