KSQL says it expects existing topic to have 2 partitions (topic has 1),

Question

I am facing below error when trying to create Stream in KSQL from an existing Kafka topic.

io.confluent.ksql.exception.KafkaTopicExistsException: A Kafka topic with the name 'test-data' already exists, with different partition/replica configuration than required. KSQL expects 2 partitions (topic has 1), and 1 replication factor (topic has 1).

Is it mandatory to have 2 partitions for creating a stream in KSQL?

It is not required to have 2 partitions to create a stream - can you share the version of KSQL that you are using and the command that you ran to create the stream? — Almog, Nov 14 '19 at 18:28
i have the same issue, even overwriting thte values on the values.yaml file of the helm chart, it does not work: configurationOverrides: "ksql.sink.partitions": "1" "ksql.sink.replicas": "3" — jacktrade, Dec 10 '19 at 15:33
Can you show your create statement? KSQL seems to say it wants to partitions in the topic — OneCricketeer, Jan 23 '20 at 06:02

score 0 · Answer 1 · answered Jun 02 '20 at 09:59

I'm guessing you're running a command such as:

CREATE TABLE FOO (<some column defs>)
  WITH (
     partitions=<some-value>, <-- explicitly setting partition count
     kafka_topic='test-data', 
     value_format='<something>'
 );

Specifically, are you explicitly setting the partition count in the WITH clause?

Neither the PARTITIONS nor the REPLICAS properties need to be set in the WITH clause when creating a TABLE or STREAM over an existing topic. These only need to be set when you wish ksqlDB to create a new topic for your data. If they are set, they must match any existing topic.

These doc pages provide more information on this subject:

https://docs.ksqldb.io/en/latest/developer-guide/create-a-stream/ https://docs.ksqldb.io/en/latest/developer-guide/create-a-table/

If this explanation does not cover your error, then please provide more information. E.g. ksql version, statements used, details of existing topics in Kafka etc.

KSQL says it expects existing topic to have 2 partitions (topic has 1),

1 Answers1