0

I execute shell script kafka-topics.sh to alter partition count from 3 to 10.

./bin/kafka-topics.sh --zookeeper xxx.xx.xxx.xx:2181/my-broker-0 --alter --topic target-topic-example-1 --partitions 10

As soon as I execute the command, Kafka cluster occurred errors.

[2018-11-20 14:45:29,954] INFO Created log for partition [target-topic-example-1,5] in /data/kafka/brokers with properties {flush.messages -> 9223372036854775807, message.timestamp.type -> CreateTime, segment.bytes -> 536870912, preallocate -> false, cleanup.policy ->    [delete], delete.retention.ms -> 86400000, segment.ms -> 604800000, min.insync.replicas -> 1, file.delete.delay.ms -> 60000, retention.ms -> 259200000, max.message.bytes -> 10000000, message.format.version -> 0.10.1-IV2, index.interval.bytes -> 4096, segment.index.bytes -> 10485760, retention.bytes -> -1, segment.jitter.ms -> 0, min.cleanable.dirty.ratio -> 0.5, compression.type -> producer, unclean.leader.election.enable -> true, message.timestamp.difference.max.ms -> 9223372036854775807, min.compaction.lag.ms -> 0, flush.ms ->          9223372036854775807}. (kafka.log.LogManager)
[2018-11-20 14:45:29,955] INFO Partition [target-topic-example-1,5] on broker 8: No checkpointed highwatermark is found for partition [target-topic-example-1,5] (kafka.cluster.Partition)
[2018-11-20 14:45:29,957] INFO Completed load of log target-topic-example-1-4 with 1 log segments and log end offset 0 in 1 ms (kafka.log.Log)
[2018-11-20 14:45:29,957] INFO Created log for partition [target-topic-example-1,4] in /data/kafka/brokers with properties {flush.messages -> 9223372036854775807, message.timestamp.type -> CreateTime, segment.bytes -> 536870912, preallocate -> false, cleanup.policy ->    [delete], delete.retention.ms -> 86400000, segment.ms -> 604800000, min.insync.replicas -> 1, file.delete.delay.ms -> 60000, retention.ms -> 259200000, max.message.bytes -> 10000000, message.format.version -> 0.10.1-IV2, index.interval.bytes -> 4096, segment.index.bytes -> 10485760, retention.bytes -> -1, segment.jitter.ms -> 0, min.cleanable.dirty.ratio -> 0.5, compression.type -> producer, unclean.leader.election.enable -> true, message.timestamp.difference.max.ms -> 9223372036854775807, min.compaction.lag.ms -> 0, flush.ms ->          9223372036854775807}. (kafka.log.LogManager)
[2018-11-20 14:45:29,958] INFO Partition [target-topic-example-1,4] on broker 8: No checkpointed highwatermark is found for partition [target-topic-example-1,4] (kafka.cluster.Partition)
[2018-11-20 14:45:29,958] INFO [ReplicaFetcherManager on broker 8] Removed fetcher for partitions target-topic-example-1-5,target-topic-example-1-4 (kafka.server.ReplicaFetcherManager)
[2018-11-20 14:45:29,958] INFO Truncating log target-topic-example-1-5 to offset 0. (kafka.log.Log)
[2018-11-20 14:45:29,958] INFO Truncating log target-topic-example-1-4 to offset 0. (kafka.log.Log)
[2018-11-20 14:45:29,961] INFO [ReplicaFetcherManager on broker 8] Added fetcher for partitions List([target-topic-example-1-5, initOffset 0 to broker BrokerEndPoint(1,My-broker02-172.cm.skp,9092)] , [target-topic-example-1-4, initOffset 0 to broker BrokerEndPoint(0,My-  broker01-172.cm.skp,9092)] ) (kafka.server.ReplicaFetcherManager)

[2018-11-20 14:45:30,062] ERROR [ReplicaFetcherThread-0-1], Error for partition [other-topic-1,3] to broker 1:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread)
[2018-11-20 14:45:30,070] ERROR [ReplicaFetcherThread-0-1], Error for partition [other-topic-2,0] to broker 1:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread)
[2018-11-20 14:45:30,070] ERROR [ReplicaFetcherThread-0-1], Error for partition [other-topic-3,3] to broker 1:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread)
[2018-11-20 14:45:30,070] ERROR [ReplicaFetcherThread-0-1], Error for partition [other-topic-4,0] to broker 1:org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request (kafka.server.ReplicaFetcherThread)

My command to change the number of partitions worked well as a result. However, I do not know what the above errors are like.

(For reference, there are a total of 10 brokers, and more than 500 topics.)

Dogil
  • 107
  • 1
  • 17
  • Sounds like broker ID 1 is offline – OneCricketeer Nov 20 '18 at 06:38
  • By the way, increasing partitions will not move existing data, just in case you weren't aware – OneCricketeer Nov 20 '18 at 06:39
  • Oh, I did not mention it, but an error occurred in broker01 that launched shell. However, the error (NotLeaderForPartitionException) occurred for 'target-topic-example-1' and is an acceptable error. – Dogil Nov 20 '18 at 06:44
  • I wonder why UnknownServerException occurred on other topics not related to target-topic-example-1. – Dogil Nov 20 '18 at 06:45
  • You're not modifying other topics at the moment, so there's no reason for those to throw any errors? Basically, unknown server is just a reclassified UnknownHostException - there's no network route that resolves to the advertised address of that broker – OneCricketeer Nov 20 '18 at 11:09
  • Previously, I have used the partition increment shell often, but this is the first time it has occurred. I used the shell today, but the error did not occur. I think I should take a look at Kafka source. Thanks, cricket! – Dogil Nov 22 '18 at 04:51
  • Your broker might have just been offline, or non responsive before... Hard to say. It might help to look at setting up some monitoring solution like JMX gathering – OneCricketeer Nov 22 '18 at 16:44

1 Answers1

0

Just altering the number of partitions isn't enough, you need to re-balance the cluster. The newer partitions are only active when the consumers are informed about these new partitions.

Please refer to my answer for process of adding partitions: https://stackoverflow.com/a/58293572/8702415

I would also recommend to conduct preferred replica election. ./bin/kafka-preferred-replica-election.sh --zookeeper localhost:9092

c0der512
  • 546
  • 5
  • 18