I am setting fetch.max.wait.ms and fetch.min.bytes for my Kafka consumer.
I am trying to understand the behavior.
Let's take an example:
- A topic with 4 partition
- 2 consumers.
- Each consumer gets 2 partitions to work with.
- fetch.min.bytes is so high that it is never met.
Does the broker have some sort of timer for each consumer, so it can know when the fetch.max.wait.ms condition is met?
How can a specific broker evaluate the amount of data that the consumer will get? this consumer might read from other partitions as well, that are not hosted on this broker.
From some experiments i made it seems that the configuration works, exactly as it described in the documentation, when the consumer works with a topic that has a single partition.
On the other hand, when a consumer is reading from multiple partitions the behavior is not clear. I see fetch requests being answered at a much higher rate than defined in fetch.max.wait.ms.
Does this difference make sense somehow? (working with MSK)
[This question i a follow up for https://stackoverflow.com/questions/72812954/kafka-msk-a-configuration-of-high-fetch-max-wait-ms-and-fetch-min-bytes-is-beh]
