How can setConsumerRebalanceListener get the calling ConsumerAwareRebalanceListener instance?

Question

Maybe this is a naive question but it somhow stuck me there for some time. Please bear with me.

I have a class DataConsumer.java that implements ConsumerAwareRebalanceListener:

@Component
public class DataConsumer implements ConsumerAwareRebalanceListener {
    @Override
    public void onPartitionsAssigned(Consumer<?, ?> consumer, Collection<TopicPartition> partitions) {
        // seek offsets based on a given timestamp
    }
    @KafkaListener(topics = "dataTopic", containerFactory = "kafkaListenerContainerFactory")
    receive(ConsumerRecord payload) {}
}

So in order for onPartitionsAssigned to work, i need to call setConsumerRebalanceListener in the kafkaListenerContainerFactory method which is defined in another class like this:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, ConsumerRecord> factory = new ConcurrentKafkaListenerContainerFactory();
    factory.getContainerProperties().setConsumerRebalanceListener(____________);
    // rest part omitted
}

My question is about this ____________ part above. What shall i put there?

In my understanding, method kafkaListenerContainerFactory is called when we initialize the @KafkaListener container in DataConsumer class, so meaning there is already an existing DataConsumer instance to hold the @kafkaLister. How can i pass that already existing DataConsumer instance to setConsumerRebalanceListener function?

All the sample code snippets I can search out are like below:

setConsumerRebalanceListener(new ConsumerRebalanceListener() {
    //override the functions
})

But isn't this creating a new instance? If I put new DataConsumer() it will lose some status in the existing instance (e.g. the timestamp to seek offsets) so this can't work.

@Gary Russell Can you help with this? Thanks! – yifei Jun 01 '20 at 11:49 — yifei, Jun 01 '20 at 11:49

score 1 · Accepted Answer · answered Jun 01 '20 at 12:56

1

You can declare DataConsumer as a @Bean (instead of using @Component) then you can inject your bean there.

However, this is the wrong mechanism to use in this case.

Implement ConsumerSeekAware instead - the container will automatically detect that your listener implements that and will call its onPartitionsAssigned.

See Seeking to a Specific Offset in the documentation.

answered Jun 01 '20 at 12:56

Gary Russell

166,535
14
146
179

Thanks Gary. If I implement `ConsumerSeekAware`, is there a way to get consumer in `onPartitionsAssigned` so I can perform `offsetsForTimes`? We use version 2.1.10 where `seekToTimestamp` is not available. – yifei Jun 02 '20 at 04:14
2.1.x is no longer supported; if you can't upgrade to a more recent supported version for some reason, you will have to use the first technique that I described above. – Gary Russell Jun 02 '20 at 13:07
Thanks Gary. I solved this with a temporary workaround, but still thinking version upgrade could be a better way for future. – yifei Jun 06 '20 at 14:31
BTW, you can leave it as a `@Component` and then `public ConcurrentKafkaListenerContainerFactory kafkaListenerContainerFactory(ConsumerAwareRebalanceListener listener) {...}` – Gary Russell Jun 06 '20 at 14:46
This perfectly solved my problem! I was thinking `kafkaListenerContainerFactory()` is required by Spring so can't be changed, but apparently param list is not fixed. Thanks Gary for pointing that out. – yifei Jun 09 '20 at 15:00

How can setConsumerRebalanceListener get the calling ConsumerAwareRebalanceListener instance?

1 Answers1