How does routing in SCS Kafka work when it comes when defines two binders for a single topic vs piping multiple consumers with a single binder

Question

I have multiple consumers that must listen to a single Kafka topic. I've found there are two ways to do it:

1. Use a binder in config per consumer Bean and specify a route to manage it via routing condition (using Spel)

2. Use only a single binder and define a single consumer Bean and use andThen() to pipe them and specify an internal condition in each method to only process the message if the condition is met.

example:

@Bean
public Consumer<Message> doStuff(){
  doA().andThen(doB());
}

private void doA(){
  if(condition)
  //do stuff
  else //ignore
}

private void doB(){
  if(condition)
  //do stuff
  else //ignore
}

Apart from the fact that in the second approach, if an exception is thrown in the first method, it would not go further to the second method (as the sequence order matters), are there any other side effects? I am interested to know if in the first approach both consumers can run in parallel and consume messages independently or because they listen to the same topic, there is no true difference as one consumer can block the other one?

Answer 1

in Kafka, multi-consumers run in different threads and consume messages independently. When multiple consumers are subscribed to the same topic and belong to the same consumer group, each consumer receives messages from different subsets of the partitions in the topic. if you use the second approach, the program will run in one thread, and its concurrency and throughput will be limited.