0

Kafkajs creates a task that at a determined frequency polls messages from kafka, making it cumbersome to implement backpressure (not impossible).

I wonder if there's an undocumented way of polling messages that allows me to implement something like this


async ()=>{
  while(true){
    const message = await consumer.poll()
    await handler(message)
  }
}
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
caeus
  • 3,084
  • 1
  • 22
  • 36
  • Not sure I understand the question. You'd use eachMessage function, not a while loop https://kafka.js.org/docs/consuming or use eachBatch, where you can better control if a consumer will heartbeat – OneCricketeer Sep 18 '22 at 15:32
  • with eachMessage or eachBatch I have to commit to a certain frequency of polling. I need a polling mechanism that polls as fast as consumer can handle messages – caeus Sep 19 '22 at 14:26
  • Polling as fast as possible is the default behavior. At least in NodeJS, where there is only a single main-event loop... If you want faster polling, you need to de-couple the processing and polling threads. – OneCricketeer Sep 19 '22 at 15:54
  • @OneCricketeer Not with kafkajs. In the run config I need to define a polling frequency, and even in the docs it is stated that we need to be careful with it, because if consumer cannot handle speed of kafkajs, event loop gets filled with unfulfilled promises – caeus Sep 19 '22 at 16:17
  • What property are you referring to in the above docs? Nothing on that page says "frequency". It you want poll at a given frequency, or implement backpressure, you need to pause+resume manually... https://kafka.js.org/docs/consuming#a-name-pause-resume-a-pause-resume – OneCricketeer Sep 19 '22 at 16:27

0 Answers0