When to use ConcurrentKafkaListenerContainerFactory?

Question

I am new to kafka and i went through the documentation but I couldn't understand anything. Can someone please explain when to use the ConcurrentKafkaListenerContainerFactory class? I have used the Kafkaconsumer class but I see ConcurrentKafkaListenerContainerFactory being used in my current project. Please explain what purpose it serves.

No prizes for guessing [how they came up with that name](https://projects.haykranen.nl/java/) — Michael, Mar 06 '19 at 12:37
ConcurrentKafkaListenerContainerFactory is from the spring framework and can be used in the spring ecosystem. KafkaConsumer is from Apache's Java sdk for Kafka. Both are just different tools/apis to implement Kafka consumers on Java. Just might differ in the capabilities provided. — Madhu Bhat, Mar 06 '19 at 12:50
Thanks @MadhuBhat for the info but that's what I wanted to know, when would I want to use ConcurrentKafkaListenerContainerFactory over kafkaconsumer and what extra capabilities it provides. — Rahul Gupta, Mar 06 '19 at 13:03

score 28 · Accepted Answer · answered Mar 06 '19 at 23:03

The Kafka consumer is NOT thread-safe. All network I/O happens in the thread of the application making the call. It is the responsibility of the user to ensure that multi-threaded access is properly synchronized. Un-synchronized access will result in ConcurrentModificationException.

If a consumer is assigned multiple partitions to fetch data from, it will try to consume from all of them at the same time, effectively giving these partitions the same priority for consumption. However in some cases consumers may want to first focus on fetching from some subset of the assigned partitions at full speed, and only start fetching other partitions when these partitions have few or no data to consume.

Spring-kafka

ConcurrentKafkaListenerContainerFactory is used to create containers for annotated methods with @KafkaListener

There are two MessageListenerContainer in spring kafka

KafkaMessageListenerContainer
ConcurrentMessageListenerContainer

The KafkaMessageListenerContainer receives all message from all topics or partitions on a single thread. The ConcurrentMessageListenerContainer delegates to one or more KafkaMessageListenerContainer instances to provide multi-threaded consumption.

Using ConcurrentMessageListenerContainer

@Bean
KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<Integer, String>>
                    kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
                            new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());
    factory.setConcurrency(3);
    factory.getContainerProperties().setPollTimeout(3000);
    return factory;
  }

It has a concurrency property. For example, container.setConcurrency(3) creates three KafkaMessageListenerContainer instances.

If you have six TopicPartition instances are provided and the concurrency is 3; each container gets two partitions. For five TopicPartition instances, two containers get two partitions, and the third gets one. If the concurrency is greater than the number of TopicPartitions, the concurrency is adjusted down such that each container gets one partition.

here is the clear example with documentation here

What is "consumerFactory()" doing? I have seen this in a few examples but it is never explained. — devo, Feb 19 '20 at 14:16
Found an example of consumerFactory here: Found an example here: https://www.javacodegeeks.com/2018/05/spring-apache-kafka-tutorial.html — devo, Feb 19 '20 at 14:30
Does that mean that each `KafkaMessageListenerContainer` creates a new consumer and runs in its own thread ? — Olivier Boissé, May 13 '22 at 10:32

score 4 · Answer 2 · answered Mar 06 '19 at 20:23

4

Kafka Consumer API is not thread safe. ConcurrentKafkaListenerContainerFactory api provides concurrent way of using Kafka Consumer API along with setting other kafka consumer properties.

answered Mar 06 '19 at 20:23

Rohit Yadav

2,252
16
18

When to use ConcurrentKafkaListenerContainerFactory?

2 Answers2