1

I have a Kafka Cluster of 3 nodes with 1 ZK and 1 Broker each.

Kafka version 0.10.1.1.

Each Topic has replication-factor=3 and min.insync.replicas=2. We have checked by issuing --describe thereafter, that the Topic(s) were successfully created with correct replication-factor and ISR.

I can see after a while few of the Topics shrunk their ISR into a Single Leader.

/opt/kafka/bin/kafka-topics.sh --describe --zookeeper zk1:2181,zk2:2181,zk3:2181 --topic topic1
Topic:topic1  PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: topic1 Partition: 0    Leader: 1       Replicas: 1     Isr: 1

/opt/kafka/bin/kafka-topics.sh --describe --zookeeper zk1:2181,zk2:2181,zk3:2181 --topic topic2
Topic:topic2  PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: topic2 Partition: 0    Leader: 1       Replicas: 1     Isr: 1

When I connect to zk Shell. I can see all Nodes are up and the controller is in broker:2

/opt/kafka/bin/zookeeper-shell.sh --describe --zookeeper zk1:2181,zk2:2181,zk3:2181
ls /brokers/ids
[0,1,2]

get /controller
{"version":1,"brokerid":2,"timestamp":"1531740996571"}

Even if I try to restart the Brokers (Only brokers or along with zookeepers), the Topics don't expand.

Any advice is highly appreciated!

Divs
  • 1,578
  • 2
  • 24
  • 51

1 Answers1

3

Looking at the topic's descriptions, it looks like these topics were created with a replication factor of 1:

ReplicationFactor:1

Having default.replication.factor=2 and min.insync.replicas=2 set in the broker configs does not prevent the creation of topics with replication factor=1.

To fix that, you can increase the replication factor of these topics using the kafka-reassign-partitions.sh tool.

To prevent this happening, you can use the Create Topic and Alter Config Policies to reject topics with an invalid replication factor.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Thanks. Will check. I have checked the `history` of the commands and the `--create` clearly showed `replication-factor=3`. – Divs Jul 17 '18 at 08:15
  • Command used for create: `/opt/kafka/bin/kafka-topics.sh --create --zookeeper zk1:2181,zk2:2181,zk3:2181 --replication-factor 3 --partitions 1 --config min.insync.replicas=2 --topic topic1` – Divs Jul 17 '18 at 08:22
  • Can you please guide me to some usages of Create Topic Policy and Alter Config Policy? – Divs Jul 17 '18 at 10:16
  • These are not available in 0.10.1. The create topic policy was added in 0.10.2 and alter configs in 0.11. Also policies are only used when creating/altering topics via the Kafka Admin APIs. If you're interested in using them, upgrade to a newer release, check the javadoc links from my answer and read the configuration description for `create.topic.policy.class.name` and `alter.config.policy.class.name` from http://kafka.apache.org/documentation/#brokerconfigs – Mickael Maison Jul 17 '18 at 10:37