1

We are using Kafka Streams via Spring Cloud Stream integration. I configured the replication factor to be used across all internal Kafka Streams topics by setting

spring.cloud.stream.kafka.streams.binder.configuration.replication.factor=${REPL_FACTOR}

It works for most repartition/changelog topics used internally by Kafka Streams. However, it looks like this setting has no effect on state store changelog topics that get created manually via Materialized#as(StoreSupplier). For those topics I can still see the replication factor is set to default 1. It's also not possible to set it using Materialized#withLoggingEnabled(Map<String, String>) because this only accepts topic-level configs (replication.factor is Streams config). Is this a known bug in Kafka Streams? I couldn't find anything. If so, is there a workaround to increase the replication factor for those changelog topics?

We are using Kafka v2.3.1 on the broker side and 2.5.0 on client side.

msilb
  • 505
  • 6
  • 18
  • I am not too familiar with streams, but you can change the default on the broker. https://kafka.apache.org/documentation/#default.replication.factor Not sure if that will help. – Gary Russell Oct 14 '20 at 14:14
  • Description says this setting is only applied for 'automatically created topics' so I guess only if [auto.create.topics.enable](https://kafka.apache.org/documentation/#auto.create.topics.enable) is set to true (which is not the case for us). – msilb Oct 15 '20 at 12:52
  • See my answer; I've added a note to the JIRA to fix the documentation. Also, what version are you using? I just tested it with 3.0.8 and it works as expected; see the edit to my answer. – Gary Russell Oct 15 '20 at 16:07

1 Answers1

1

Starting with version 2.4, the AdminClient can now set the replication factor to -1 in NewTopic, meaning that the default.replication.factor should be used when creating topics - KIP-464.

However, it appears that Kafka Streams does not currently use this feature; there is an open issuee KAFKA-8531 for this.

You can set the replication factor for internal topics using

StreamsConfig.REPLICATION_FACTOR_CONFIG)

https://kafka.apache.org/documentation/#replication.factor

The replication factor for change log topics and repartition topics created by the stream processing application.

Since you are setting that, via the binder config, it should work as expected.

EDIT

What version of spring-cloud-stream are you using? I just tested with 3.0.8 and it works as expected.

spring.cloud.stream.kafka.streams.binder.configuration.replication.factor: 3

2020-10-15 12:03:55,601 ERROR [kafka-stre] o.a.k.s.p.i.StreamThread:673 - stream-thread [kafka-streams-inventory-processor-b8d07a5a-f3c4-476a-a265-119163d2acb7-StreamThread-1] Encountered the following unexpected Kafka exception during processing, this usually indicate Streams internal errors: org.apache.kafka.streams.errors.StreamsException: Could not create topic kafka-streams-inventory-processor-inventory-counts-changelog.

Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179