What is a listener container in Spring for Apache Kafka?

Question

I understood that to make a method to be the target of Kafka message listener, I have to mark this method with the @KafkaListener annotation. This annotation lets specify by containerFactory element the KafkaListenerContainerFactory.

Below there are snippets by Baeldung Spring Kafka Tutorial.

KafkaConsumerConfig.java

private ConsumerFactory<String, String> consumerFactory(String groupId) {
    Map<String, Object> props = new HashMap<>();
    ...
    return new DefaultKafkaConsumerFactory<>(props);
}

private ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(String groupId) {
    ConcurrentKafkaListenerContainerFactory<String, String> factory =
            new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory(groupId));
    return factory;
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> fooKafkaListenerContainerFactory() {
    return kafkaListenerContainerFactory("foo");
}

MessageListener.java

@KafkaListener(
    topics = "${message.topic.name}", 
    groupId = "foo", 
    containerFactory = "fooKafkaListenerContainerFactory")
public void listenGroupFoo(String message) {
    System.out.println("Received Message in group 'foo': " + message);
    ...
}

What I didn't understand is why we need a factory of listeners container. What is a listener container? What happen when a method is annotated in that way?

Gary Russell · Accepted Answer · 2020-09-22T15:36:56.100

30

A listener "container" is a Spring concept across multiple technologies (JMS, RabbitMQ, Kafka, AWS, etc, etc).

The listener is defined as a POJO bean method and is a property of the container (the container "contains" the listener).

The container is responsible for interacting with the broker to receive messages and invoke your listener method with each message, or a batch of messages, depending on the listener type.

This means your application code does not have to deal with the mechanics of interacting with the broker and you can concentrate on your business logic only.

The framework discovers any @KafkaListener methods and uses the factory to create a container for each one.

edited Sep 22 '20 at 15:36

answered Sep 22 '20 at 15:17

Gary Russell

166,535
14
146
179

Thanks for reply. Can I say, at high level, that container creation correspond to register a listener on the specified topics for consumers? – Vin Sep 22 '20 at 16:55
5

Yes; but concurrency comes into play; if your topic(s) have more than one partition, and the container concurrency is > 1, you get that number of consumers that can call your listener concurrently with records from their subscribed partition(s). In this case we have one `ConcurrentMessageListenerContainer` with `n` "child" `KafkaMessageListenerContainer`s, but they all invoke the same listener instance. – Gary Russell Sep 22 '20 at 17:04
Now it's clear. However I noticed another thing. If I put the same `@KafkaListener` annotation (same topic and same groupId) on two different methods, I notice that only the latter is invoked. It seem that the annotation on the second methods "hides" the first one. It's only about casuality or there is a reason? – Vin Sep 22 '20 at 17:24
5

That would happen if the topic has only one partition - Kafka only allows one consumer per partition unless they are in different groups. When there is only one partition, the second consumer would be created but would be assigned 0 partitions - you can see that in the logs. – Gary Russell Sep 22 '20 at 17:32
@GaryRussell, where is the consumer concept between container and listener? Seems there is no room for `consumer`? – FrankGod Apr 21 '22 at 09:37
3

Don't ask new questions in comments on old answers; it doesn't help people find answers. The listener container starts a thread for each `concurrency` (default 1); each thread creates a consumer and interacts with its API to get records and pass them to the listener, either one at a time, or in a batch, depending on the type of listener. – Gary Russell Apr 21 '22 at 13:27
@GaryRussell instead of using ```@KafkaLIstener```, I am using ```class MyMessageListener implements MessageListener``` and wanted to bring concurrency.Can you let me know how can I achieve that – Utkarsh Saraf Jun 08 '22 at 11:25
1

Don't ask new questions in comments; it doesn't help people find questions/answers. See https://docs.spring.io/spring-kafka/docs/current/reference/html/#concurrency – Gary Russell Jun 08 '22 at 13:17

What is a listener container in Spring for Apache Kafka?

1 Answers1