1

I'm using a kafka spring consumer that is under group management.

I have the following code in my consumer class

public class ConsumerHandler implements Receiver<String, Message>, ConsumerSeekAware {

    @Value("${topic}")
    protected String topic;

    public ConsumerHandler(){}


    @KafkaListener(topics = "${topic}")
    public Message receive(List<ConsumerRecord<String, Message>> messages, Acknowledgment acknowledgment) {

        for (ConsumerRecord<String, Message> message : messages) {
            Message msg = message.value();                    
            this.handleMessage(any, message);
        }

        acknowledgment.acknowledge();

        return null;
    }



    @Override
    public void registerSeekCallback(ConsumerSeekCallback callback) {
    }

    @Override
    public void onPartitionsAssigned(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) {

            for (Entry<TopicPartition, Long> pair : assignments.entrySet()) {
                TopicPartition tp = pair.getKey();
                callback.seekToEnd(tp.topic(),tp.partition());
            }           

    }

    @Override
    public void onIdleContainer(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) {}
}

This code works great while my consumer is running. However, sometimes the amount of messages being processed is too much and the messages stack up. I've implemented concurrency on my consumers and still sometimes there's delay in the messages over time.

So as a workaround, before I figure out why the delay is happening, I'm trying to keep my consumer up to the latest messages.

I'm having to restart my app to get partition assigned invoked so that my consumer seeks to end and starts processing the latest messages.

Is there a way to seek to end without having to bounce my application?

Thanks.

Garuuk
  • 2,153
  • 6
  • 30
  • 59

2 Answers2

1

As explained in the JavaDocs and the reference manual, you can save off the ConsumerSeekCallback passed into registerSeekCallback in a ThreadLocal<ConsumerSeekCallback>.

Then, you can perform arbitrary seek operations whenever you want; however, since the consumer is not thread-safe, you must perform the seeks within your @KafkaListener so they run on the consumer thread - hence the need to store the callback in a ThreadLocal.

In version 2.0 and later, you can add the consumer as a parameter to the @KafkaListener method and perform the seeks directly thereon.

public Message receive(List<ConsumerRecord<String, Message>> messages, Acknowledgment acknowledgment,
    Consumer<?, ?> consumer) {

The current version is 2.1.6.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
0

I have never found or seen a fixed solution for this kind of problem. The way I do is to boost performance as high as possible base on the amount of messages to be processed and Kafka parameters.

Let's say if you have a shopping online app then you can control the upper bound of the number of transactions per day, said N. So you should make the app work well in the scenario where 1.5*N or 2*N transactions will need to sync to Kafka cluster. You keep this state until a day your shopping app reaches a new level and you will need to upgrade your Kafka system again. For shopping online app there are a special high number of transactions in promotion or mega sales days so what you prepare for your system is for these days.

Quang Vien
  • 308
  • 2
  • 10