Managing Kafka Topic with spring

Question

We are planning to use Kafka for queueing in our application. I have some bit of experience in RabbitMQ and Spring.

With RabbitMQ and Spring, we used to manage queue creation while starting up the spring service.

With Kafka, I'm not sure what could be the best way to create the topics? Is there a way to manage the topics with Spring.

Or, should we write a separate script which helps in creating topics? Maintaining a separate script for creating topics seems a bit weird for me.

Any suggestions will be appreciated.

kafka 1.1.0 has `auto.create.topics.enable` set to true by default, if it is ok for your production requirements you are already set :D — Paizo, Jun 18 '18 at 12:35
In general I do not see issues by keeping it enable since 99% of the time kafka is configured to be reachable only by local network and on top of that you can add authentication, so it depends on your architecture and security requirements — Paizo, Jun 18 '18 at 12:46
I'd prefer disabling it to prevent the app from creating wrong/unnecessary topic accidentally. If it's on local network, a pre-prod app startup might be able to access the cluster? Depends on your NW setting/security, but disabling might be wise choice IMHO. — Divs, Aug 02 '18 at 11:12

score 11 · Accepted Answer · answered Jun 18 '18 at 12:54

In spring it is possible to create topics during the start of the application using beans:

@Bean
public KafkaAdmin admin() {
    Map<String, Object> configs = new HashMap<>();
    configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
            StringUtils.arrayToCommaDelimitedString(kafkaEmbedded().getBrokerAddresses()));
    return new KafkaAdmin(configs);
}

@Bean
public NewTopic topic1() {
    return new NewTopic("foo", 10, (short) 2);
}

Alternatively you can write your own create topics by autowiring the AdminClient, so for instance reading the list from an input file or specify advanced properties such as partition numbers:

@Autowired
private KafkaAdmin admin;
//...your implementation

Also note that since Kafka 1.1.0 auto.create.topics.enable is enabled by default (see Broker configs).

For more information refer to the spring-kafka docs

Spring Boot auto configures a `KafkaAdmin` so you only need to add the `NewTopic` `@Bean`s in a boot application. — Gary Russell, Jun 18 '18 at 13:59

jumping_monkey · Answer 2 · 2020-08-12T06:49:01.540

To automatically create a Kafka topic in Spring Boot, only this is required:

@Bean
public NewTopic topic1() {
    return new NewTopic("foo", 10, (short) 2);

    //foo: topic name
    //10: number of partitions
    //2: replication factor
}

The Kafka Admin is being automatically created and configured by Spring Boot.

Version 2.3 of Spring Kafka introduced a TopicBuilder class, to make building topics fluent and more intuitive:

@Bean
public NewTopic topic(){
    return TopicBuilder.name("foo")
        .partitions(10)
        .replicas(2)
        .build();
}

Managing Kafka Topic with spring

2 Answers2