How to wait for Kafka respone in a Jest test with KafkaJS?

Question

Consider this test, in which a message is sent from the test to topic 'out', and the tested code is expected to consume it and reply by sending a message to topic 'in'. In order to pass, I want to make sure a message was sent to topic 'in'.

it('...', async () => {
  /* initialize kafkaConsumer and kafkaProducer here */

  async function someCallback() {
    // ...
  }

  await kafkaConsumer.subscribe({ topic: 'in', fromBeginning: true })
  await kafkaConsumer.run({ eachMessage: someCallback })

  await kafkaProducer.send({ topic: 'out', messages: [{ key: '1', value: '2' }] })

  // How do I block here until someCallback is called?
})

I read about using done but I can't have that while the test itself is defined async, which I need in order to use await. Is there a different way I'm not aware of?

hi do you mind sharing more code like setting up producer, topic, kafka server etc. that you have used for this. — Kalyan Chavali, Mar 31 '21 at 20:09
@KalyanChavali Take a look at the answer I have just now posted below — chikchakchok, Apr 02 '21 at 11:16

score 4 · Answer 1 · edited Dec 10 '21 at 00:31

You can take a look at how we test KafkaJS itself for some inspiration. For example, here's a basic consumer test.

We are really not doing anything fancy, just adding the messages to an array from within the eachMessage callback, and then await a promise that periodically checks if we have reached the expected number of messages. Something like this:

it('consumes messages', async () => {
  const messages = [{ value: 'hello world' }]
  const consumedMessages = []

  consumer.run({
    eachMessage: ({ message }) => {
      consumedMessages.push(message);
    }
  })

  await producer.send({ topic, messages })

  await waitFor(() => consumedMessages.length === messages.length)
})

Where waitFor is essentially a function that returns a promise and starts a setTimeout that checks the predicate and resolves the promise when the predicate is true (or rejects if it hits a timeout).

Some gotchas to keep in mind:

Use a new groupId on each run, so that multiple runs don't interfere with each other.
Use a new topic on each test run for the same reason.
If you produce messages before your consumer has joined the group and subscribed to the topic, those messages won't show up by default. Either subscribe with fromBeginning: true or wait for your consumer to have subscribed and joined the group before producing (the instrumentation events emit an event on group join that you can wait for in the same way as we wait for the messages to be consumed).

how can i import this waitFor? couldn't find any example anywhere for that — Kalyan Chavali, Mar 31 '21 at 17:45
also do i need testhelpers for creating a kafka test in my project? — Kalyan Chavali, Mar 31 '21 at 20:55
@KalyanChavali you can implement `waitFor` yourself using `Promise.race` between a `timeout` promise and the consumption promise like that: https://pastebin.com/Aiu66rdH — chikchakchok, Apr 02 '21 at 11:22
@KalyanChavali you can also use the wait-for-expect module if you are using jest for your testing library — Avocado, Dec 08 '21 at 20:36

score 2 · Accepted Answer · answered Apr 02 '21 at 11:15

After some time with Tommy Brunn's answer, I found a few bugs and I ended up with this:

export const waitForKafkaMessages = async (
  kafka: Kafka,
  messagesAmount: number,
  topic: string,
  fromBeginning: boolean,
  groupId: string,
): Promise<KafkaMessage[]> => {
  const consumer: Consumer = kafka.consumer({ groupId })
  await consumer.connect()
  await consumer.subscribe({ topic, fromBeginning })

  let resolveOnConsumption: (messages: KafkaMessage[]) => void
  let rejectOnError: (e: Error) => void

  const returnThisPromise = new Promise<KafkaMessage[]>((resolve, reject) => {
    resolveOnConsumption = resolve
    rejectOnError = reject
  }).finally(() => consumer.disconnect()) // disconnection is done here, reason why is explained below

  const messages: KafkaMessage[] = []
  await consumer.run({
    autoCommit: false,
    eachMessage: async ({ message, partition, topic }) => {
      try {
        // eachMessage is called by eachBatch which can consume more than messagesAmount.
        // This is why we manually commit only messagesAmount messages.
        if (messages.length < messagesAmount) {
          messages.push(message)

          // +1 because we need to commit the next assigned offset.
          await consumer.commitOffsets([{ topic, partition, offset: (Number(message.offset) + 1).toString() }])
        }

        if (messages.length === messagesAmount) {
          // I think we should be able to close the connection here, but kafkajs has a bug which makes it hang if consumer.disconnect is called too soon after consumer.run .
          // This is why we close it in the promise's finally block

          resolveOnConsumption(messages)
        }
      } catch (e) {
        rejectOnError(e)
      }
    },
  })

  return returnThisPromise
}

Hmm but how do we always produce always different topic? I produce from my function and topic is coming from process.env. Consumer uses the same topic... Did you decide to pass that new topic to all your functions or are you using source connectors so you don't worry about that? — Djordje Nikolic, Apr 02 '21 at 14:07
I just ended up making new topics with ID of my data model and making sure it is unique across all my tests. In my main function i have process.env.TOPIC || post.id. Be careful tho because of topic naming conventions you might get an error. Also @chikchakchok very good approach. I used same method as u. THANKS A LOT!!!! — Djordje Nikolic, Apr 02 '21 at 15:46

How to wait for Kafka respone in a Jest test with KafkaJS?

2 Answers2