How do I create a Kafka topic on the fly / on startup for the producer to send to?

Question

I'm starting to use the Confluent .NET library for Kafka and am attempting to implement a pattern I used to use against Azure Service Bus, to create the topic upon startup of the producer application (create if not exists). How would this be done in the Kafka API's, and can it be done?

This would allow the topics to be part of source control and configured in the automated release processes rather than manually set up per topic/environment. Also, I'd prefer that my developers not have to go to each Kafka instance / environment and configure them first to match.

If I can't do it this way, I'll have to bake it into bash scripts in the release process, but would prefer it in the startup code.

Answer 1

You could enable the cluster-wide configuration auto.create.topics.enable.

This will automatically create a topic if a new producer tries to send data to a topic that does not exist yet.

However, be aware of the following:

• the topic will be created with default settings on replication, number of partitions and retention. Make sure to change those default setting as required. Anyway, all automatically created topic will have the identical configuration.
• typos in the topic name configuration in the producer code can lead to unwanted creation of topics.

Alternatively, you can make use of the AdminClient API. An example is shown here:

static async Task CreateTopicAsync(string bootstrapServers, string topicName)
{
    using var adminClient = new AdminClientBuilder(new AdminClientConfig
    {
        BootstrapServers = bootstrapServers
    }).Build();

    try
    {
        await adminClient.CreateTopicsAsync(new TopicSpecification[] 
        {
            new TopicSpecification 
            {
                Name = topicName,
                ReplicationFactor = 1,
                NumPartitions = 1
            }
        });
    }
    catch (CreateTopicsException e)
    {
        Console.WriteLine($"An error occured creating topic {e.Results[0].Topic}: {e.Results[0].Error.Reason}");
    }
}