0

I have 2 Kafka topics A and B. I want that every time when my consumer starts up, it goes to topic A first. Topic A contains information about topic B, and I'll depend on that information to fetch data from topic B.

So essentially I need to read topic A first before going to topic B, and I just need to do this once upon every restart of my program.

Things I can think of like:

@KafkaListener(topics = {"A" , "B"})

Or:

@KafkaListener(topics = "A")
public void receive() {}

@KafkaListener(topics = "B")
public void receive() {}

Both don't guarantee the sequence of reading.

How can I enforce my program to read topic A first, and only go to topic B after finishing the latest updates on topic A?

yifei
  • 561
  • 5
  • 18
  • For people who kept voting this question as not "focused", can you show me how i should divide it into multiple sub-qns to satisfy your criteria? Or what are the different qns you think it contains? i mean, seriously, the subject line and the last sentence of the description are not the same, but it doesn't mean it contains 2 questions right? – yifei May 28 '20 at 10:51
  • I didn't vote, but 'needs more focus' doesn't necessarily mean it is asking more than one question, it can also mean you're question is too broad (as in the solution could take a whole article or book). In any case, I think you can't solve this easily. Kafka topics are by their nature asynchronous, so there is no real order. If you want order, you probably need to use a pull-based mechanism. – Mark Rotteveel May 28 '20 at 12:37
  • Thanks Mark, got the solution from Russell below. It is not an essay-like answer, and it's exactly what i need. That's why i feel people may be too hurry clicking "needs more focus". – yifei May 29 '20 at 08:04

1 Answers1

1

Use something like this...

@KafkaListener(id = "bReceiver", autoStartup = "false, topics = "B")
public void receive() {}

Set the idleEventInterval and add

@Autowired
KafkaListenerEndpointRegistry registry;

@EventListener(condition = "event.listenerId == 'aReceiver`)
public void eventListener(ListenerContainerIdleEvent event) {
    this.registry.getListenerContainer("aReceiver").stop(() -> { });
    this.registry.getListenerContainer("bReceiver").start();
}
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Hi Russell, while attempting this approach I hit some new issue can you kindly help to take a look? Thank you! https://stackoverflow.com/questions/62091106 – yifei May 29 '20 at 17:14