0

So, to keep it short:

Procedure: Frontend -\> start node -\> consume messages -\> send result back to frontend

Once I start a node and the first request is processed, the results get sent back in an instant.

However if I send the second request (\<20 seconds), It takes very long for the consumer to join the group.

I get the Debug Message:

{
    "level":"DEBUG",
    "timestamp":"2023-02-22T14:43:54.714Z",
    "logger":"kafkajs",
    "message":"\[Connection\] Request JoinGroup(key: 11, version: 5)",
    "broker":"bmmc02cq2x3md6p.bechtle.net:9092",
    ...
}

Then I am left hanging for \~10 seconds, then I get the response message from Kafka:

{
    "level":"DEBUG",
    "timestamp":"2023-02-22T14:44:23.245Z",
    "logger":"kafkajs",
    "message":"\[Connection\] Response JoinGroup(key: 11, version: 5)",
    "broker":"bmmc02cq2x3md6p.bechtle.net:9092",
    "cli...
}

Does anyone know what's going on ?

sachuverma
  • 559
  • 6
  • 27
Pixel2203
  • 1
  • 1
  • do you start the entire consumer on every request from front end? if that is the case then yes it will take time because Kafka will rebalance the group and that would take time – Neenad Feb 23 '23 at 09:09
  • What do you mean with 'start the entire consumer' ? I disconnect once it has read all messages from the topic and returned that result. If then a second request gets send, the consummer has to connect again, read msg... and disconnects then itself – Pixel2203 Feb 23 '23 at 12:47
  • Kafka consumers are generally long-running tasks they keep polling the broker and work on new messages, in case of task failure broker will rebalance. If you are connecting every time to the broker the broker as a new process broker has to reassign the task again. read this: https://sergiuoltean.com/2020/08/19/kafka-consumer-rebalance/ – Neenad Feb 23 '23 at 21:47
  • Try setting `session.timeout.ms` to higher value https://kafka.apache.org/documentation/#consumerconfigs_session.timeout.ms or use static membership https://kafka.apache.org/documentation/#static_membership – Neenad Feb 23 '23 at 21:50
  • I also read about static membership, but there seems to be no static membership in KafkaJS – Pixel2203 Feb 24 '23 at 06:02

1 Answers1

0

In KafkaJS there’s two options you could try

  1. sessionTimeout

Timeout in milliseconds used to detect failures. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance

  1. rebalanceTimeout

The maximum time that the coordinator will wait for each member to rejoin when rebalancing the group

And even after this your app is slow I would suggest write a long running JS consumer store the result in DB and for frontend fetch the data from DB

https://kafka.js.org/docs/consuming

Neenad
  • 861
  • 5
  • 19