1

I am making a Chat application, using Socket io for sending and receiving messages. I want to use Kafka as a message broker, I am using unique Socket io rooms for each pair of users for privacy, I am expecting the Kafka consumer to subscribe to the newly created room but Kafka isn't letting me do that when the consumer is running. I am using KafkaJS, Typescript.

KafkaJSNonRetriableError: Cannot subscribe to topic while consumer is running at Object.subscribe (E:\Tech\realtime-chat\server\node_modules\kafkajs\src\consumer\index.js:136:13) at consumeMessages (E:\Tech\realtime-chat\server\src\sockets\socket.ts:46:20) at E:\Tech\realtime-chat\server\src\sockets\socket.ts:33:9 at processTicksAndRejections (node:internal/process/task_queues:96:5) { retriable: false, helpUrl: undefined, [cause]: undefined }

  socket.on("join", async (data) => {
    const topic = generateRoomId(data.sender, data.recipient);
    console.log(topic);
    socket.join(topic);

    admin
      .connect()
      .then(() =>
        admin.createTopics({
          topics: [{ topic }],
        })
      )
      .then(() => {
        console.log(`Topic "${topic}" created successfully`);

        consumeMessages(topic);
        handlePrivateChat(topic);
      })
      .catch((error) => {
        console.error("Error creating topic:", error);
      });
  });


 const consumeMessages = async (topic: string) => {
    await consumer.subscribe({ topic });
    await consumer.run({
      eachMessage: async ({ message }) => {
        socket.emit("message", message.value?.toString());
      },
    });
  };

  const handlePrivateChat = async (topic: string) => {
    socket.on("message", async (data) => {
      const message: Message = {
        sender: data.sender,
        recipient: data.recipient,
        content: data.content,
        timestamp: new Date(),
      };
      await producer.send({
        topic,
        messages: [{ value: JSON.stringify(message) }],
      });
    });
  };

I want the consumers to subscribe to newly created room on runtime, is there any other alternative to this?

Karteek Ati
  • 43
  • 5
  • I suppose, you should move `await consumer.subscribe({ topic });` outside of `consumeMessages`. Subscription should be done once at the script start. – quentino May 25 '23 at 12:40
  • You only have one consumer instance. You cannot make it subscribe/run again without stopping the first subscription... Also, Apache Pulsar has built-in websocket support, so you don't really need SocketIO and Kafka – OneCricketeer May 26 '23 at 08:07

0 Answers0