13

Or maybe even a way to delay the message received by the consumer. I need to make a function call in nodejs after every 90s, so I want to add a delay of 90s for every kafka message

Samarth Juneja
  • 766
  • 1
  • 8
  • 21
  • 1
    Okay, so set a sleep or timer interval in the consumer poll loop (or the producer, but I would recommend consumer)... Have you not tried this? – OneCricketeer May 08 '18 at 01:06
  • I want the delay to happen at the kafka end and not at my main project end. I think the timer interval and sleep that you've suggested, would be inside the main project – Samarth Juneja May 08 '18 at 04:59
  • You have two ends of the Kafka pipe. I would typically suggest getting real time events in as quick as possible, then consumers can be as slow as they want within the topic retention period – OneCricketeer May 08 '18 at 05:03
  • It's not the topic retention I am concerned about, it's the delay in receiving in the message itself. That is what I need. I need a function call after a 90s timeout. My problem is I can't seem to find that functionality in kafka – Samarth Juneja May 08 '18 at 05:25
  • I'm sorry to tell you that doesn't exist in Kafka. You need to implement that yourself as an external client – OneCricketeer May 08 '18 at 05:28
  • For example, in Spark Streaming you can set a 90s batch window size. I'm sure in similar frameworks you can define time windows upon which an aggregate function can be applied, if that's what you're trying to do – OneCricketeer May 08 '18 at 05:30

1 Answers1

4

You could make use of the KafkaProducer configuration linger.ms which is described as:

"[...] adds a small amount of artificial delay — that is, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. [...]"

This configuration defaults to 0 and can be set to any long value.

Depending on the amount of data that your producer sends to a topic, you might also want to increase the configuration batch.size. Otherwise, if this size limit it reached before the delay of linger.ms is over, the KafkaProducer will send the messages before the 90 seconds.

Keep in mind that if you increase linger.ms you may also want to increase delivery.timeout.ms. According to its documentation:

"The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms."

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
  • This works in conjunction with batch.size, if that is reached early it will not wait for linger.ms. So to achieve this you'll have to keep your batch size very large such that it waits for linger.ms every time, but if your scale is large this could become potentially a bottle neck – krishan May 05 '23 at 02:33