Pass list of topics from application yml to KafkaListener

Question

I have the following application.yml:

service:
  kafka:
    groupId: 345
    consumer: 
      topics: 
        -
          name: response    
    producer: 
      topics: 
        -
          name: request1 
          num-partitions: 5
          replication-factor: 1 
        -
          name: request2 
          num-partitions: 3
          replication-factor: 1

How can I access the list of topic names using spel for passing to KafkaListener annotation?

@KafkaListener(topics = "#{'${service.kafka.consumer.topics.name}'}", containerFactory = "kafkaListenerContainerFactory")
public void receive(String payload, @Header(KafkaHeaders.RECEIVED_TOPIC)String topic) {

see my answer [here](https://stackoverflow.com/a/52402517/3224238) At the time of writing i did not found a more elegant way — Paizo, Oct 10 '18 at 13:18
@Paizo I cannot use comma separate topics since I need other parameters related to each topic — cppcoder, Oct 10 '18 at 13:20
Then I would go with `@ConfigurationProperties` for that and instead of @KafkaListener go with the `@Bean` way to define it. Unless you find a pretty way with the `SPEL` expression (I didn't) than I would be happy to see how as well :) — Paizo, Oct 10 '18 at 13:24
With `@ConfigurationProperties` you can use projection on the collection of topics; see my answer. — Gary Russell, Oct 10 '18 at 15:22

Gary Russell · Answer 1 · 2021-03-17T21:33:53.373

Use configuration properties and collection projection...

@ConfigurationProperties("service.kafka.producer")
@Component
public class ConfigProps {

    List<Topic> topics = new ArrayList<>();

    public List<Topic> getTopics() {
        return this.topics;
    }

    public void setTopics(List<Topic> topics) {
        this.topics = topics;
    }

    @Override
    public String toString() {
        return "ConfigProps [topics=" + this.topics + "]";
    }

    public static class Topic {

        private String name;

        private int numPartitions;

        private short replicationFactor;

        public String getName() {
            return this.name;
        }

        public void setName(String name) {
            this.name = name;
        }

        public int getNumPartitions() {
            return this.numPartitions;
        }

        public void setNumPartitions(int numPartitions) {
            this.numPartitions = numPartitions;
        }

        public short getReplicationFactor() {
            return this.replicationFactor;
        }

        public void setReplicationFactor(short replicationFactor) {
            this.replicationFactor = replicationFactor;
        }

        @Override
        public String toString() {
            return "Topic [name=" + this.name + ", numPartitions=" + this.numPartitions + ", replicationFactor="
                    + this.replicationFactor + "]";
        }

    }

}

and

@SpringBootApplication
public class So52741016Application {

    public static void main(String[] args) {
        SpringApplication.run(So52741016Application.class, args);
    }

    @KafkaListener(groupId = "${service.kafka.groupId}", topics = "#{configProps.topics.![name]}")
    public void listener(String in) {

    }

    @Bean
    public SmartLifecycle createTopics(KafkaAdmin admin, ConfigProps props) {
        return new SmartLifecycle() {

            @Override
            public int getPhase() {
                return Integer.MIN_VALUE;
            }

            @Override
            public void stop() {
            }

            @Override
            public void start() {
                try (AdminClient client = AdminClient.create(admin.getConfig())) {
                    CreateTopicsResult createTopics = client.createTopics(props.topics.stream()
                        .map(t -> new NewTopic(t.getName(), t.getNumPartitions(), t.getReplicationFactor()))
                        .collect(Collectors.toList()));
                    createTopics.all().get();
                }
                catch (Exception e) {
//                  e.printStackTrace();
                }
            }

            @Override
            public boolean isRunning() {
                return false;
            }

            @Override
            public void stop(Runnable callback) {
            }

            @Override
            public boolean isAutoStartup() {
                return true;
            }
        };
    }

}

and

2018-10-10 11:20:25.813 INFO 14979 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer : partitions assigned: [request1-4, request2-0, request1-0, request2-1, request1-1, request2-2, request1-2, request1-3]

Of course, this is only the producer topics, but you can handle them all this way.

This works fine together with ConfigurationProperties: `@KafkaListener(topics = "${spring.kafka.topic.creation}")` — julianm, Mar 17 '21 at 21:22

Pass list of topics from application yml to KafkaListener

1 Answers1

Linked