2

I am writing a Kafka Streams application, and I would like to include two application id in this application, but I keep getting error saying that "Topology with no input topics will create no stream threads and no global thread, must subscribe to at least one source topic or global table." Could you please let me know where I made a mistake? Thank you so much!

public class KafkaStreamsConfigurations {
    ...
    @Bean(name = KafkaStreamsDefaultConfiguration.DEFAULT_STREAMS_CONFIG_BEAN_NAME)
    @Primary
    public KafkaStreamsConfiguration kStreamsConfigs() {
        Map<String, Object> props = new HashMap<>();
        setDefaults(props);
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "default");
        return new KafkaStreamsConfiguration(props);
    }

    public void setDefaults(Map<String, Object> props) {...}

    @Bean("snowplowStreamBuilder")
    public StreamsBuilderFactoryBean streamsBuilderFactoryBean() {
        Map<String, Object> props = new HashMap<>();
        setDefaults(props);
        ...
        props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 0);
        props.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 1);

        Properties properties = new Properties();
        props.forEach(properties::put);
        StreamsBuilderFactoryBean streamsBuilderFactoryBean = new StreamsBuilderFactoryBean();
        streamsBuilderFactoryBean.setStreamsConfiguration(properties);
        return streamsBuilderFactoryBean;
    }
}

Here is my application class.

public class SnowplowStreamsApp {
    @Bean("snowplowStreamsApp")
    public KStream<String, String> [] startProcessing(
        @Qualifier("snowplowStreamBuilder") StreamsBuilder builder) {
                KStream<String, String>[] branches = builder.stream(inputTopicPubsubSnowplow, Consumed
            .with(Serdes.String(), Serdes.String()))
            .mapValues(snowplowEnrichedGoodDataFormatter::formatEnrichedData)
            .branch(...);
        return branches;
    }
}
Shan
  • 177
  • 3
  • 12

1 Answers1

3

Name your factory bean DEFAULT_STREAMS_BUILDER_BEAN_NAME instead of snowplowStreamBuilder - otherwise, the default factory bean will be started with no defined streams.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Thanks for your help. My application should be able to refer this bean. In this case, what should I put in the @Qualifier section? Also, since I will have two applications in the same project, how could I have a second StreamsBuilderFactoryBean for the second application? Thank you! – Shan Jun 04 '21 at 20:15
  • You don't need a qualifier if there is only one `StreamBuilderFactoryBean`. Your second question is not clear - if it's a second application, each one will have a single instance of the bean. – Gary Russell Jun 04 '21 at 20:19
  • Sorry for the confusion. I am following a similar step from https://stackoverflow.com/questions/51733039/kafka-streams-with-spring-boot. I would like to define two StreamsBuilderFactoryBean, and I could have two application id in this case. Thank you again. – Shan Jun 04 '21 at 20:24
  • 2
    Oh, I see - they changed the Kafka Streams client since that answer to disallow a Topology with no input topics. With that answer, the default SBFB was unused but, back when that answer was written, that was allowed. With modern versions of `kafka-streams`, you will need to use the default SBFB for one of your topologies and your custom one for the other. OR, you can disable the creation of the default SBFB by NOT having `@EnableKafkaStreams` on any of your `@Configuration` classes. – Gary Russell Jun 04 '21 at 20:39
  • Thank you sooooo much! You helped a lot! I noticed they disallowed a topology with no input topics, but I did not recognize the annotation part. Thank you! – Shan Jun 04 '21 at 20:49