17

In kafka producer settings, what is the expected behavior if you have linger.ms set to 0 and a non zero batch.size? How long will the producer wait for batch.size to be reached before sending the messages? Will it wait forever till the size of messages is less than specified batch size OR since linger.ms is zero, it will not do any batching and just send every request?

Vivek Agarwal
  • 301
  • 1
  • 2
  • 8

1 Answers1

23

No, this does not mean that the producer will wait for the batch to become full. The producer will send half-full batches and even batches with just a single message in them.

Therefore, setting the batch size too large will not cause delays in sending messages; it will just use more memory for the batches. Setting the batch size too small will add some overhead because the producer will need to send messages more frequently.

By default (linger.ms=0), the producer will send messages as soon as there is a sender thread available to send them, even if there’s just one message in the batch.

Hope it helps.

Thanks.

dossani
  • 1,892
  • 3
  • 14
  • 23
  • 1
    So essentially with linger.ms=0 there will be no batching irrespective of the batch.size parameter's value? – Vivek Agarwal Jul 25 '18 at 17:43
  • 1
    linger.ms gives upper bound on the delay for batching. By default(linger.ms =0), there's no additional wait time. A batch size of zero will disable batching entirely. – dossani Jul 25 '18 at 18:07
  • 9
    Ok, so let's say we have defined a batch size of 10kb, then messages can be buffered in this batch till 10kb is reached or a sender thread becomes available. If a sender thread becomes available at 5kb, it will just send these 5kb worth of messages. Otherwise at 10kb it will anyways try to send explicitly. If we had linger.ms,then producer will wait at least that amount of time even if sender thread became available earlier and batch size is not reached. Correct me if I am wrong @dossani – Vivek Agarwal Jul 25 '18 at 20:30
  • Yes, you are right. linger.ms introduces a bit of latency. – dossani Jul 25 '18 at 20:38
  • 2
    Latency using linger.ms is introduced to increase the chances of batching. By introducing a small delay, we can send more messages in a batch thereby improving throughput and less latency as there are less no of request sent over to Apache Kafka – Gautham Honnavara Oct 19 '20 at 03:07
  • 1
    What happens if application tries to write more messages when the buffer is full, but a sender thread is still not available? – Lahiru Chandima Feb 05 '21 at 08:59
  • 1
    Ofcouse this answer would help, it exactly copied and pasted from lafka definitive guide. Doesnt mean it wouldnt help – Hiresh Feb 05 '22 at 04:48
  • @LahiruChandima a Timeout exception is thrown if buffer is full and "max.timeout.ms" has passed – Pintu Feb 21 '23 at 10:28