0

I want to implement retry logic while consuming from Kafka topic using KafkaJS, so basically, I will have 2 topics main-topic and retry-topic and I will

read from -> main-topic
if processing fails | -> retry topic

so is it a bad practice to use one consumer for listening from both topics(both main and retry), as kafka allows to listen from multiple topics using same consumer.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Kaneki21
  • 1,323
  • 2
  • 4
  • 22

1 Answers1

1

It's not a bad practice at all.

The only problem you may run into using one consumer is that the topics may need differ configurations (connection settings, deserializer, etc). In that case, you can create two separate Consumer instances rather than one subscribing to both.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Thanks @OneCricketeer, suppose I have 3-4 sets of `topics` (main and retry) and I'm using `nodejs` single thread. Won't it affect the event loop if I have 6-8 different `kafka consumer instance` ? – Kaneki21 Dec 26 '22 at 13:40
  • 1
    Not that I know of. Polling is a blocking operation, so the thread would alternate between the two (or more). You could also look at the child_process module for spawning multiple OS node processes – OneCricketeer Dec 26 '22 at 16:45