12

What's the difference? Can the term KafkaConsumer and KafkaListener be used interchangeably?

George
  • 2,820
  • 4
  • 29
  • 56
  • They are not interchangeable in Java code. KafkaConsumer is a class from the Kafka client library which provides the API for applications to receive messages. KafkaListener is an annotation applied to a method so Spring Kafka will invoke it to process a message. – Chin Huang Apr 03 '19 at 14:27
  • So what does each do? – George Apr 03 '19 at 14:28

1 Answers1

16

The @KafkaListener is a high level API for the ConcurrentMessageListenerContainer, which spawns several internal listeners around KafkaConsumer.

The difference is that that KafkaConsumer API is pollable on demand when you call its poll() whenever you need. The listener abstraction is about to have an infinite loop around that poll() and it produces messages for records whenever they appear from the poll(). We have there a task executor which runs a logic like this:

        while (isRunning()) {
            try {
                pollAndInvoke();
            }
            catch (@SuppressWarnings(UNUSED) WakeupException e) {
                // Ignore, we're stopping
            }
            catch (NoOffsetForPartitionException nofpe) {
                this.fatalError = true;
                ListenerConsumer.this.logger.error("No offset and no reset policy", nofpe);
                break;
            }
            catch (Exception e) {
                handleConsumerException(e);
            }
            catch (Error e) { // NOSONAR - rethrown
                Runnable runnable = KafkaMessageListenerContainer.this.emergencyStop;
                if (runnable != null) {
                    runnable.run();
                }
                this.logger.error("Stopping container due to an Error", e);
                wrapUp();
                throw e;
            }
        }

The KafkaConsumer.poll() is called in that pollAndInvoke();.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • 2
    When I set the property `spring.kafka.listener.concurrency` to 4, will 4 consumers be instantiated, or is it 4 listeners? – George Apr 03 '19 at 14:49
  • 1
    The listener is going to be the same for all the consumers. It is stateless, so it is fully fine do not spawn more listeners. The concurrency fully depends on the number of partitions you are going to have from your topics configuration for the listener. So, if you have in the end only one partition, the concurrency configuration doesn't matter. – Artem Bilan Apr 03 '19 at 14:52
  • What does this property specifically do (`spring.kafka.listener.concurrency`)? I'm using `ConcurrentMessageListenerContainer`, turns out in order to increase concurrency, I've to use `setConcurrency` of the factory and this property doesn't help. – Ashwani Agarwal May 18 '22 at 09:29
  • If you use container factory manually, then Spring Boot backs off and those configuration properties have no effects – Artem Bilan May 18 '22 at 11:52
  • Does it spring Kafka spawns n consumers if listener is configured with topic of n partitions even without declaring setConcurrency ? I didn't understand why and when this configuration has no effects ? – Pintu Mar 10 '23 at 12:29
  • The doc says setConcurrency creates multiple consumer based on parameter and yes if concurrency > partition number , then the delta consumer won't be created – Pintu Mar 10 '23 at 12:36
  • 1
    So, do you still have questions? The framework spans not more consumers, then minimum of `concurrency` and `partition number`. So, if your concurrency is `1`, but `10` partitions - still one consumer. If opposite: concurrency is `10`, but `1` partition, still one consumer. – Artem Bilan Mar 10 '23 at 14:06