List Kafka Topics via Spring-Kafka

Question

We would like to list all Kafka topics via spring-kafka to get results similar to the kafka command:

bin/kafka-topics.sh --list --zookeeper localhost:2181

When running the getTopics() method in the service below, we get org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

Configuration:

@EnableKafka
@Configuration
public class KafkaConfig {
    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:2181");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, 
            StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, 
            StringDeserializer.class);
        return new DefaultKafkaConsumerFactory<>(props);
}

Service:

@Service
public class TopicServiceKafkaImpl implements TopicService {
    @Autowired
    private ConsumerFactory<String, String> consumerFactory;

    @Override
    public Set<String> getTopics() {
        try (Consumer<String, String> consumer = 
            consumerFactory.createConsumer()) {
            Map<String, List<PartitionInfo>> map = consumer.listTopics();
            return map.keySet();
    }
}

Kafka is up and running and we can send messages from our app to a topic succesfully.

score 9 · Answer 1 · answered Jul 30 '19 at 19:20

You can list topics like this using Admin Client

    Properties properties = new Properties();
    properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

    AdminClient adminClient = AdminClient.create(properties);

    ListTopicsOptions listTopicsOptions = new ListTopicsOptions();
    listTopicsOptions.listInternal(true);

    System.out.println("topics:" + adminClient.listTopics(listTopicsOptions).names().get());

Gary Russell · Accepted Answer · 2018-11-28T22:04:57.327

6

You are connecting to Zookeeper (2181) instead of Kafka (9092 by default).

The Java kafka clients no longer talk directly to ZK.

edited Nov 28 '18 at 22:04

answered Nov 28 '18 at 21:43

Gary Russell

166,535
14
146
179

Awesome. All of the tutorials / examples that I've seen talk to Zookeeper. – Paul Croarkin Nov 28 '18 at 22:03
While pointing out the code-flaw, this doesn't really answer how to get the list of topics with spring-kafka – OneCricketeer Nov 29 '18 at 04:10
?? While I haven't tested it myself, the `KafkaConsumer.listTopics()` Javadocs says `Get metadata about partitions for all topics that the user is authorized to view. This method will issue a remote call to the server.`. **all topics the user is authorized to view** seems pretty clear to me. – Gary Russell Nov 29 '18 at 04:23
And `assignment()` says `Get the set of partitions currently assigned to this consumer.`. – Gary Russell Nov 29 '18 at 04:26
Fair enough, though I do personally find it odd to go through the consumer and [fetcher API to get said information](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L278-L285) – OneCricketeer Nov 29 '18 at 05:12

OneCricketeer · Answer 3 · 2018-11-29T05:10:44.037

4

kafka-topics --list is a shell script that just is a wrapper around kafka.admin.TopicCommand class, where you can find the method you are looking for

Alternatively, you can also use the AdminClient#listTopics method

edited Nov 29 '18 at 05:10

answered Nov 29 '18 at 04:09

OneCricketeer

179,855
19
132
245

I agree it's odd; however, I suppose if you already have a consumer, it would save opening a new connection for this purpose. I wouldn't open a new consumer just for this, though. I also wouldn't get involved with the scala client mentioned in your first sentence. – Gary Russell Nov 29 '18 at 13:33

List Kafka Topics via Spring-Kafka

3 Answers3