0

I am trying to get my consumer to dynamically update its consumption.

Let me give you a more concrete example using animals. Imagine that I have a pet store, every topic is a type of animal (e.g. dogs, cats, fish). The primary responsibility of my Kafka consumer is to grab whatever log/record/message we have in Kafka and store them into a database.

Suppose that my consumer is actively consuming on dogs and cats topics and everything works fine, now there is a new type of animal coming into the store and a new topic is generated in the Kafka cluster. How do I notify my consumer that a new topic has been added?

I have two proposals and I want to see which one do you think is better? Or if there is a better 3rd option, please let me know.

1.) Producer sends a http request to consumer and let consumer knows that producer is about to create a new topic so consumer can act accordingly. The problem with this approach is that, there is a race condition. There is a chance that consumer will try to consume before the topic is even created. (I actually just discovered that if I have auto.topic.creation.enable set to true, the race condition isn't really a problem.)

2.) Create an extra topic called topic_updates in the Kafka cluster. So whenever producer has successfully committed a message to the Kafka cluster, it broadcasts the news through this topic_updates, maybe a simple string will do. Consumer is actively listening to this topic updates.

3.) I don't know, ideally I wish that Kafka can emit an event whenever a new topic is created.

Thank you in advance

mofury
  • 644
  • 1
  • 11
  • 23

2 Answers2

3

Consumer is able to find out the new-created topics automatically, and you could simply subscribe all topics by invoking consumer.subscribe(Pattern.compile(".*"));

Could lower down metadata.max.age.ms to have consumer aware of the new topics more quickly.

amethystic
  • 6,821
  • 23
  • 25
  • It's not working for me. As I have tried `kafka-console-consumer --bootstrap-server localhost:9092 --whitelist "mytopics.*" --consumer-property metadata.max.age.ms=5000` and `kafka-console-producer --broker-list localhost:9092 --topic mytopics.1234`. I still get `[2019-12-20 15:00:59,080] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {mytopics.1234=LEADER_NOT_AVAILABLE}` (org.apache.kafka.clients.NetworkClient) – addlistener Dec 20 '19 at 08:01
0

You can use the new KafkaAdminClient and somehow monitor the list of topics and check for new additions. Here's a sample code that gives you the list of topics (excluding the internal topics):

Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
KafkaAdminClient kafkaAdminClient = (KafkaAdminClient) AdminClient.create(properties);
ListTopicsResult listTopicResult = kafkaAdminClient.listTopics();
System.out.println(listTopicResult.names().get().toString());
vahid
  • 1,118
  • 9
  • 13