How can I effectively bind my @KafkaListener to ConcurrentKafkaListenerContainerFactory?

Question

I hit this scenario which appears strange to me:

So basically I have defined two @KafkaListener in one class:

@KafkaListener(id = "listener1", idIsGroup = false, topics = "data1", containerFactory = "kafkaListenerContainerFactory")
    public void receive(){}

@KafkaListener(id = "listener2", idIsGroup = false, topics = "data2", containerFactory = "kafkaListenerContainerFactory2")
    public void receive(){}

Their id, topics, containerFactory are different, and each of them relies on a different ConcurrentKafkaListenerContainerFactory as defined in another class:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> factory = new ConcurrentKafkaListenerContainerFactory();
    factory.setConsumerFactory(consumerFactory("group1", "earliest"));
    factory.setAutoStartup(false);
    return factory;
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> kafkaListenerContainerFactory2() {
    ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> factory = new ConcurrentKafkaListenerContainerFactory();
    factory.setConsumerFactory(consumerFactory("group2", "latest"));
    factory.setAutoStartup(true);
    return factory;
}

@Bean
public ConsumerFactory<String, ConsumerRecord> consumerFactory(String groupId, String offset) {
    Map<String, Object> config = new HashMap<>();
    // dt is current timestamp in millisecond (epoch)
    config.put(ConsumerConfig.GROUP_ID_CONFIG, groupId + "-" + dt);
    config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, offset);
    // other config omitted
    return new DefaultKafkaConsumerFactory<>(config);
}

So what I expect to see (and what I want to achieve) are:

Only listener2 will auto-start because factory.setAutoStartup(true)
Listener2 will start with group.id "group2" and auto.offset.reset "latest"
Later when listener1 starts via some event listener, it will start with group.id "group1" and auto.offset.reset "earlist"

However, only the 1st is actually guaranteed. Listener2 can start with either {group2 + latest} or {group1 + earliest}. And later when listener1 starts to consume data, it will just reuse the config of listener2 (I can see the same group id which contains a timestamp is printed twice in my log)

My question is, why the group ID and offset config for listener2 are randomly picked while autoStartup is not? And why listener1 will reuse the config of listener2?

Gary Russell · Accepted Answer · 2020-05-30T13:30:00.387

1

It's because consumerFactory is a singleton @Bean and the arguments are ignored on the second call.

Add @Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE) to the factory get a new bean each time.

However, you don't need any of this, you can simply set the groupId property on the annotations and avoid all this extra definition.

You can also control autoStartup on the annotation (since 2.2).

EDIT

To answer the question in the comment below...

groupId = "#{'${group.id}' + T(java.time.Instant).now().toEpochMilli()}"

however, if you want a unique group id; this is more reliable...

groupId = "#{'${group.id}' + T(java.util.UUID).randomUUID()}"

edited May 30 '20 at 13:30

answered May 29 '20 at 17:39

Gary Russell

166,535
14
146
179

Thanks Gary! Adding `@Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)` works well. If I want to keep it singleton, is there a way to set groupId with the current timestamp on the annotation? Something like `@KafkaListener(id = "${topic}", groupId="${group.id}" + Instant.now().toEpochMilli())`? (This line is not working of course) – yifei May 30 '20 at 08:41
Thanks a lot Gary! Just as a note to myself and people come across this question in the future, this use case of SpEL is well documented here https://docs.spring.io/spring/docs/3.0.x/reference/expressions.html section 6.4 – yifei May 30 '20 at 17:07
And the [javadocs for `groupId` here](https://github.com/spring-projects/spring-kafka/blob/96efd3fa554dc83b28c8e9013fe226c9af48951a/spring-kafka/src/main/java/org/springframework/kafka/annotation/KafkaListener.java#L165-L172). Probably not much has changed, but that SpEL documentation link is very old (spring 3.0). The current SpEL documentation is [here](https://docs.spring.io/spring/docs/5.2.6.RELEASE/spring-framework-reference/core.html#expressions). – Gary Russell May 30 '20 at 17:30

How can I effectively bind my @KafkaListener to ConcurrentKafkaListenerContainerFactory?

1 Answers1

Linked