1

I'm trying to fetch messages from Kafka using kafkajs I want to catch all messages into "messages" (an array) config - the variable which hold the configs Here is the function I'm using:

const { Kafka, logLevel } = require('kafkajs')
async function consume_messages(config) {
    const kafka = new Kafka({
        logLevel: logLevel.INFO,
        brokers: [config.broker],
        ssl: true,
        sasl: {
            mechanism: [config.mechanism], 
            username: config.Username,
            password: config.Password
        },
    })
    const topic = config.client_id
    const consumer = kafka.consumer({
        groupId: 'my-group', fromBeginning: true
    })
    await consumer.connect();
    await consumer.subscribe({
        topics: [topic],
        fromBeginning: true
    })
    let messages = []
    await consumer.run({
        eachMessage: async ({ message }) => {
             messages.push(message)
             console.log('RECEIVED MESSAGE', JSON.parse(message.value), message.offset);
          }
        })
       
    })
    await consumer.disconnect();
    return messages ;
}

when I run it I get nothing the "messages" stays empty However, when I removed the await from "await consumer.run({" -> "consumer.run({" it worked but only after the entire script ended.

How can I force it to run and to wait for all messages to be fetched?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Gil Allen
  • 1,169
  • 14
  • 24
  • 1
    Your consumer doesn't know how much time it should work, instead it's fired, works for some small period of time, and then is almost instantly disconnected. If your app is a backend, make the `await consumer.disconnect()` call a shutdown hook. If your app is just a kafka consumer -- let it work for some time, f.e. paste `await sleep(5 min)` before the `await consumer.disconnect()`. – star67 Sep 11 '22 at 16:57
  • You appear to be trying to block a Kafka consumer. Ideally, you don't do this and instead write callbacks into the location that actually needs the records rafter than using `return`. More importantly, Kafka topics should be thought of as endless, so don't "wait til the end" – OneCricketeer Sep 11 '22 at 17:14
  • I added a delay function just before "await consumer.disconnect()" And removed the "await " from it await delay(10000) consumer.disconnect() the delay function is: function delay(time) { return new Promise(resolve => setTimeout(resolve, time)); } And it just wait 10 sec... and nothing happens... – Gil Allen Sep 12 '22 at 06:49
  • Sadly I have added delay of 20 seconds after the run. Not ideal let messages = [] await consumer.run({ eachMessage: async ({ topic, partition, message }) => { messages.push(JSON.parse(message.value)) console.log(`Total messages so far : ${messages.length}\n`); } }) await delay(20000) The delay function: function delay(time) { console.log(`\nwaiting ${time} seconds....\n`) return new Promise(resolve => setTimeout(resolve, time)); } – Gil Allen Sep 12 '22 at 11:19

1 Answers1

0

The only thing that worked for me was to add a pause after the run() finction. I must say I'm not happy about this solution since I'm forcing my side to wait fixed 5 seconds regardless to the results.

I'll be happy for a better solutions - but here it is: I have added a delay function:

    function delay(time) {
    return new Promise(resolve => setTimeout(resolve, time));
}

And then , after the run "await consumer.run(..." and before "consumer.disconnect()" I called it Like this:

    await consumer.run({
        eachMessage: async ({ message }) => {
             messages.push(message)
             console.log('RECEIVED MESSAGE', JSON.parse(message.value), message.offset);
          }
        })
           
        })
/*Here is where the code waits for messages to be pulled  */
       await delay(5000)
       consumer.disconnect();
Gil Allen
  • 1,169
  • 14
  • 24
  • Hi. Did you find a better solution? – BIndu_Madhav Jan 06 '23 at 12:23
  • nope.... I'm working with many APIs and this was the 1st time I hade to pull data from kafka.. For my needs - I wish they never used kafka... – Gil Allen Jan 08 '23 at 07:50
  • i'm finding the same issue - essentially the `await consumer.run(...)` does not actually block, it seems to return immediately, and so the disconnect runs immediately after and then your function returns nothing – Peter McIntyre May 03 '23 at 08:17