INCOMPLETE_SOURCE_TOPIC_METADATA error in KafkaStreams

Question

I writing KafkaStreams application and set the maximum.num.threads as one. I've three source topics and have 6,8,8 partitions respectively. Currently running this streamtopology with 4 instances so 4 running streams threads.

I'm getting INCOMPLETE_SOURCE_TOPIC_METADATA in one of my kafka topics. I found below code from github throwing this error and trying to understand the code

    final Map<String, InternalTopicConfig> repartitionTopicMetadata = new HashMap<>();
    for (final InternalTopologyBuilder.TopicsInfo topicsInfo : topicGroups.values()) {
        for (final String topic : topicsInfo.sourceTopics) {
            if (!topicsInfo.repartitionSourceTopics.keySet().contains(topic) &&
                !metadata.topics().contains(topic)) {
                log.error("Missing source topic {} during assignment. Returning error {}.",
                    topic, AssignorError.INCOMPLETE_SOURCE_TOPIC_METADATA.name());
                return new GroupAssignment(
                    errorAssignment(clientMetadataMap, topic,
                        AssignorError.INCOMPLETE_SOURCE_TOPIC_METADATA.code())
                );
            }
        }
        for (final InternalTopicConfig topic : topicsInfo.repartitionSourceTopics.values()) {
            repartitionTopicMetadata.put(topic.name(), topic);
        }
    }

My questions:

1. Is this error comes because of partition mismatch on Kafka topics, or TopicsInfo is not available at the time (Think like Kafka group lost its access to the Kafka topic)?

2. what is meant by topicsInfo.repartitionSourceTopics call?

Answer 1

The error means that Kafka Streams could not fetch all required metadata for the used topics and thus cannot proceed. This could happen if a topic does not exist (ie, not created yet or miss spelled), or if the corresponding metadata was not broadcasted to all brokers yet (for this case, the error should be transient). Note that metadata is updated asynchronously and thus there might be some delay until all brokers learn about a new topic.