0

Does the byte retention policy apply to replica partitions or leader partitions or both? In a multi-node cluster, with all brokers configured configured with different retention policies, it seems obvious that the partitions for which a given broker is a leader will be subject to the local byte retention policy, but what about the partitions for which the given broker is a replica? Are they subject to the same policy?

For example, a 2 node cluster with single topic and two partitions.

  • Broker A is the leader for partition 1 and contains the replica of partition 2
  • Broker B is the leader for partition 2 and contains the replica of partition 1
  • Broker A has a byte retention policy of 1 byte
  • Broker B has a byte retention policy of 2 bytes

Will broker A retain 1 byte of of both partitions it hosts? Likewise, will broker B retain 2 bytes of both partitions it hosts?

David Corley
  • 710
  • 5
  • 17
  • why would you have different retention policy on different nodes in your cluster? – Anil Gupta Jul 05 '15 at 07:28
  • @AnilGupta , the original use case was for a cluster with heterogenous storage capacity, where we wanted to use 80% of _whatever_ the locally available disk was (using a convoluted calculation involving topic counts,partition counts, replication,etc) for Kafka storage. If the partition leader was on a bigger disk than that of the replica machine, it could cause issues. We've since re-written the log.retention.bytes such that it first determines the minimum disk space available for a given node in the cluster, and sets the byte retention for all of the nodes in the cluster based on this. – David Corley Jul 07 '15 at 08:27

1 Answers1

0

Yes as per my cluster which contains 3 nodes it impact all the machine in cluster.

bin/kafka-topics.sh --zookeeper 172.29.xx.xx:2181 --alter --topic topicname --config retention.ms=2100000
Andrey
  • 485
  • 3
  • 17
Karunakar Singh
  • 1
  • 2