3

I have 4 ConsumerFactory listeners that are reading from 4 different topics like this:

@KafkaListener(
      id = "test1",
      topicPattern  = "test.topic1",
      groupId = "pp-test1")
  public void listenTopic1(ConsumerRecord<String, String> record) {
    System.out.println("Topic is: " + record.topic());   
  }

But we'll have 50 topics and I want to set up atleast 25 listeners for betetr performance. How can I do this dynamically instead of manually writing 25 listeners?

user10751899
  • 147
  • 1
  • 9

2 Answers2

3

You cannot create @KafkaListeners programmatically, only discrete listener containers (with a custom listener).

You can do it by programmatically creating a child application context for each listener.

EDIT

@SpringBootApplication
public class So53715268Application {

    public static void main(String[] args) {
        ConfigurableApplicationContext context = SpringApplication.run(So53715268Application.class, args);
        for (int i = 0; i < 2; i++) {
            AnnotationConfigApplicationContext child = new AnnotationConfigApplicationContext();
            child.setParent(context);
            child.register(ChildConfig.class);
            Properties props = new Properties();
            props.setProperty("group", "group." + i);
            props.setProperty("topic", "topic" + i);
            PropertiesPropertySource pps = new PropertiesPropertySource("listenerProps", props);
            child.getEnvironment().getPropertySources().addLast(pps);
            child.refresh();
        }
    }

}

and

@Configuration
@EnableKafka
public class ChildConfig {

    @Bean
    public Listener listener() {
        return new Listener();
    }

}

and

public class Listener {

    @KafkaListener(id = "${group}", topics = "${topic}")
    public void listen(String in) {
        System.out.println(in);
    }

}

and

: partitions assigned: [topic0-0]
: partitions assigned: [topic1-0]

Note that, if you are using Spring Boot, the child config class and listener must be in a different package to the main app (and not a sub-package either).

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Thanks. Do you have an example code of it can be done? – user10751899 Dec 11 '18 at 19:25
  • Hi, the code you wrote has some problems. If I copy it, it tells me that addLast() is not applicable for PropertiesPropertySource. How should I change it? – Usr Apr 03 '19 at 12:22
  • You must have something wrong; that code is from a working Spring Boot app. `child.getEnvironment().getPropertySources()` returns a `MutablePropertySources` which, indeed, has `addLast()`. – Gary Russell Apr 03 '19 at 12:46
0

Let's say your listener method is part of a Springboot app called Listener. Read topic name from configuration application yml:

@KafkaListener(
  topics = "${ai.kafka.consumer.topic-name}")
public void listenTopic1(ConsumerRecord<String, String> record) {
System.out.println("Topic is: " + record.topic());   
}

And then deploy as many Listener as you want, each with a different value of topic-name in application.yml. (If you're using Docker container, that's easier).

Gianx
  • 61
  • 7