1

My topic has 3 partitions (in 3 different brokers). I want to send message to particular partition 1 (add partition id or specify a key in my message). Assuming the partition 1 became full of disk, does an alternative partition (partition 2 or 3 in this case) be chosen for this message?

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
Hieu Doan
  • 115
  • 1
  • 7

1 Answers1

1

It is not the topic that is running out of disk but only the entire broker. If the broker runs out of disk it will shut itself down and you will not be able to send to or receive data from it.

As you are specifically pointing to a partition, it now depends if you have a replication-factor value larger than 1. In case you have replicated your topic partitions, the partition leader would switch from the dead broker to another broker with that has a partition in-sync.

If you do not have replication of more than 1 your producer would end up getting an OutOfMemory error and the data it not produced to the alternative partitions 2 or 3.

To make sure that this never happens you can apply the volume based retention policies that is available in Kafka if you set your cleanup.policy to delete and set the retention.bytes such that with the number of topics/replicas you will not exceed your storage limit.

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
  • what it be if I don't have replica of this partition? – Hieu Doan Oct 07 '20 at 06:06
  • Than your producer will get something like an OutOfMemory Exception. – Michael Heil Oct 07 '20 at 06:08
  • So no matter how many other partition of the same topic are available, if I continuously write to a dead partition (run out of disk), it will cause an exception and no alternative partition to serve this message? – Hieu Doan Oct 07 '20 at 06:24
  • 1
    yes, this is correct. How would Kafka now to which partition it shoule write, 2 or 3, or both? And also, Kafka has some guarantees on the ordering of messages within a partition. If it now writes data to another partition although the client wanted to write it to partition 1 it will break this ordering guarantees. – Michael Heil Oct 07 '20 at 06:34
  • Yes, Thank u a lot. – Hieu Doan Oct 07 '20 at 06:47