1

As per LINGER_MS_DOC in ProducerConfig java class:

"The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay; that is, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle's algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get "BATCH_SIZE_CONFIG" worth of records for a partition it will be sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting "LINGER_MS_CONFIG=5" for example, would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of load."

I searched for a suggested value for linger.ms but nowhere found a higher value suggested for this. Most of the places 5 ms is mentioned for linger.ms.

For testing, I have set "batch.size" to 16384 (16 KB) and "linger.ms" to 60000 (60 seconds) as per doc I felt if I send a message of size > 16384 bytes then the producer will not wait and send the message immediately, but I am not observing the same behavior.

I am sending events of size > 16384 bytes but it still waits for 60 seconds. Am I missing to understand the purpose of "batch.size"? My understanding of "batch.size" and "linger.ms" is that whichever meets first the messages/batch will be sent.

In this case, if it is going to be the minimum wait time and do not give preference to "batch.size" then I guess setting a high value for linger.ms is not right.

Here is the kafka properties used in yaml:

  producer:
    properties:
      acks: all
      retries: 5
      batch:
        size: 16384
      linger:
        ms: 10
      max:
        request:
          size: 1046528
hsingh
  • 661
  • 5
  • 26
  • Are you applying any compression that might reduce the message size? – Michael Heil Nov 09 '20 at 12:21
  • No compression is applied. – hsingh Nov 09 '20 at 12:58
  • Is ot possible to share a minimal reproducable example? – Michael Heil Nov 15 '20 at 21:21
  • I have added the YAML config used for kafka producer. Till now I observed that producer never waits more than linger.ms time and it never sends a request if total message size is less than batch.size and elapsed time is also less than linger.ms. The basic question is can Kafka producer sends a request if batch.size is achieved but elapsed time is less than linger.ms? – hsingh Nov 16 '20 at 04:28

0 Answers0