2

I am using Spring Cloud stream with Kafka binder. To disable the auto-create topics I referred to this- How can I configure a Spring Cloud Stream (Kafka) application to autocreate the topics in Confluent Cloud?. But, it seems that setting this property is not working and the framework creates the topics automatically.

Here is the configuration in application.properties

spring.cloud.stream.kafka.binder.auto-create-topics=false

Here is the startup log

2021-06-25 09:22:46.522  INFO 38879 --- [pool-2-thread-1] o.a.k.clients.consumer.ConsumerConfig    : ConsumerConfig values: 
    allow.auto.create.topics = true
    auto.commit.interval.ms = 5000
    auto.offset.reset = latest
    bootstrap.servers = [localhost:9092]

Other details-

  1. Spring boot version: 2.3.12.RELEASE
  2. Spring Cloud Stream version: Hoxton.SR11

Am I missing anything in this configuration?

Swapnil
  • 801
  • 3
  • 19
  • 42

1 Answers1

5

spring.cloud.stream.kafka.binder.auto-create-topics=false

That property configures the binder so that it will not create the topics; it does not set that consumer property.

To explicitly set that property, also set

spring.cloud.stream.kafka.binder.consumer-properties.allow.auto.create.topics=false
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Great! That worked for the consumer. Is there any way to disable this auto-create for the producer? I tried spring.cloud.stream.kafka.binder.producer-properties.allow.auto.create.topics but such property do not exist and the producer is auto-creating the missing topic – Swapnil Jun 25 '21 at 15:02
  • I don't see a similar property in `ProducerConfig` but I don't see the behavior you observe, I get a bunch of `[Producer clientId=producer-1] Error while fetching metadata with correlation id xx : {source-out-0=UNKNOWN_TOPIC_OR_PARTITION}` followed by a `TimeoutException` after 60 seconds. – Gary Russell Jun 25 '21 at 15:11
  • I can see that error ```[Producer clientId=producer-1] Error while fetching metadata with correlation id 1 : {Spring-Cloud-Producer=LEADER_NOT_AVAILABLE}``` But, it still creates the missing topic. Properties I am using are- ```spring.cloud.stream.source=producer-stream-source``` and ```spring.cloud.stream.bindings.producer-stream-source-out-0.destination=Spring-Cloud-Producer``` – Swapnil Jun 25 '21 at 15:18
  • 1
    Sorry - you are correct, my broker had auto create disabled `auto.create.topics.enable=false`. I see it is created if the broker allows it; strange that there is not a similar property for producers. Maybe reach out to the kafka folks on one of their mailing lists? – Gary Russell Jun 25 '21 at 15:22
  • Sorry, but how Kafka team will help me on this? Earlier I used spring Kafka where I did not face such an issue. So with the Spring cloud stream, I was hoping, there might be some configuration. The good thing is this won't be an issue in our non-prod & prod env as the auto-create topic is disabled. – Swapnil Jun 25 '21 at 16:14
  • `>Sorry, but how Kafka team will help me on this?` - I mean you should ask them why there is not a producer property equivalent to the consumer property. Why can't you disable `auto.create.topics.enable` on your test broker? – Gary Russell Jun 25 '21 at 16:16
  • Sure. I will disable auto-create on test brokers. How can I get in touch with those guys? Is there any different tag on Stackoverflow for that? – Swapnil Jun 25 '21 at 16:19
  • 1
    Google `kafka mailing list`. Or, open a JIRA issue https://issues.apache.org/jira/projects/KAFKA/ – Gary Russell Jun 25 '21 at 16:25
  • Note that `allow.auto.create.topics` property might be set for the specific consumer (through the `spring.cloud.stream.kafka.bindings..consumer.configuration` section) rather than for all consumers of the binder (in the `spring.cloud.stream.kafka.binder.consumerProperties` section). – Ilya Serbis Jul 20 '23 at 11:38