Kafka topics as chatroom identifiers

Question

I am interested in a messaging chat app where different chatroom can be possible. I try to understand if Kafka topics can be used as chatroom identifiers from Kafka conceptual point of view?

As chatrooms are normally created in a runtime then it means that

Kafka topics should be created in a runtime. Is it possible?
Users should be able to subscribe to the new topics (chatrooms) as soon as they are created. Is it possible?
There will be a lot of chatrooms. Is it bad or ok for Kafka to have huge number of topics?

While this is "possible", it is a bad design, particularly for a consumer perspective. You'd be better using an actual database for storing the chats rather than seeking and consuming the entire topic each and every time a client will open the chatroom. For example, LinkedIn + Facebook/WhatsApp might use a message-queue such as Kafka for actually "sending a chat message", but that will be consumed **and persisted** somewhere else (Facebook used to use HBase/Cassandra for this). When the chat is loaded, the database is queried, not the topic (refer CQRS pattern for details) — OneCricketeer, Jun 23 '22 at 19:48
And with that design, you truly do not need "one topic per chat" since the consumer can read from one or few `chats` topic, and compute a `room_id`, then write to the respective tables. — OneCricketeer, Jun 23 '22 at 19:52

score 1 · Answer 1 · edited Jun 23 '22 at 19:46

Kafka topics should be created in a runtime. Is it possible?

Yes. Kafka AdminClient provides a programmatic API for administrative functionality, including listing, creating and deleting topics.

We can use createTopics to create new topics:

Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
AdminClient admin = AdminClient.create(props);


CreateTopicsResult newTopic = admin.createTopics(Collection.singletonList(new NewTopic(TOPIC_NAME, NUM_PARTITIONS, REP_FACTOR)));

admin.close(Duration.ofSeconds(30));

The above code comes from Kafka: The Definitive Guide 2nd Edition, Chapter 5 Managing Apache Kafka Programmatically, which discusses a lot about AdminClient.

For spring-kafka, the KafkaAdmin provides createOrModifyTopics() method.

Users should be able to subscribe to the new topics (chatrooms) as soon as they are created. Is it possible?

Yes. We can use regular expressions in .subscribe():

The expression can match multiple topic names, and if someone creates a new topic with a name that matches, a rebalance will happen almost immediately and the consumers will start consuming from the new topic.

For example, to subscribe to all test topics, we can call

consumer.subscribe(Pattern.compile("test.*"));

There will be a lot of chatrooms. Is it bad or ok for Kafka to have huge number of topics?

It's okay: Can I have 100s of thousands of topics in a Kafka Cluster?

Kafka topics as chatroom identifiers

1 Answers1