Separate Kafka listener for each topic using annotations in spring kafka

Question

My application is listening to multiple kafka topics. Right now, my listener looks like below, and my.topics in property file contains the list of comma separated kafka topics

@KafkaListener(topics = ["#{'\${my.topics}'.split(',')}"], groupId = "my.group", containerFactory = "myKafkaFactory")
fun genericMessageListener(myRequest: MyRequest, ack: Acknowledgment) {
   //do Something with myRequest
    ack.acknowledge()
}

My ConcurrentKafkaListenerContainerFactory is

@Bean
fun myKafkaFactory(): ConcurrentKafkaListenerContainerFactory<String, MyRequest> {
    val factory = ConcurrentKafkaListenerContainerFactory<String, MyRequest>()
    factory.consumerFactory = DefaultKafkaConsumerFactory(configProps(), StringDeserializer(), MyRequestDeserializer())
    factory.containerProperties.ackMode = ContainerProperties.AckMode.MANUAL
    return factory
}

Is there a way i can create a separate consumer for each topic dynamically so that when I add one more topic to the list in my.topics, spring will automatically creates a separate consumer for that topic.

The problem I am facing now is, if something goes wrong with one of the messages in any of the topics, messages in other topics are also getting impacted.

On a high level, i am looking for something like this

@Bean
fun myKafkaFactory(): ConcurrentKafkaListenerContainerFactory<String, MyRequest> {
    val factory = ConcurrentKafkaListenerContainerFactory<String, MyRequest>()
    factory.consumerFactory = DefaultKafkaConsumerFactory(configProps(), StringDeserializer(), MyRequestDeserializer())
    factory.containerProperties.ackMode = ContainerProperties.AckMode.MANUAL
    factory.isOneConsumerPerTopic(true)
    return factory
}

so that factory.isOneConsumerPerTopic(true) will ensure a separate consumer is created for each topic in the Array.

I did go through How to create separate Kafka listener for each topic dynamically in springboot?. I am looking a bit more 'cleaner' solution. :)

score 1 · Answer 1 · answered Nov 15 '19 at 14:12

You can add a custom PartitionAssignor to the Kafka consumer configuration.

Set the container concurrency to (at least) the number of topics and have your assignor assign the partitions for each topic to a specific consumer(s).

/**
 * This interface is used to define custom partition assignment for use in
 * {@link org.apache.kafka.clients.consumer.KafkaConsumer}. Members of the consumer group subscribe
 * to the topics they are interested in and forward their subscriptions to a Kafka broker serving
 * as the group coordinator. The coordinator selects one member to perform the group assignment and
 * propagates the subscriptions of all members to it. Then {@link #assign(Cluster, Map)} is called
 * to perform the assignment and the results are forwarded back to each respective members
 *
 * In some cases, it is useful to forward additional metadata to the assignor in order to make
 * assignment decisions. For this, you can override {@link #subscription(Set)} and provide custom
 * userData in the returned Subscription. For example, to have a rack-aware assignor, an implementation
 * can use this user data to forward the rackId belonging to each member.
 */
public interface PartitionAssignor {

You could start with the AbstractPartitionAssignor.

See the spring-kafka documentation.

When listening to multiple topics, the default partition distribution may not be what you expect. For example, ...

Separate Kafka listener for each topic using annotations in spring kafka

1 Answers1