Multiple StreamListeners to same Topic with Spring Cloud Stream connected to Kafka

Question

I have Spring Boot app I am using Spring Cloud Stream connecting to Kafka. I'm trying to set up two separate stream listener methods to same kafka topic.

@StreamListener("countries")
    @SendTo("aggregated-statistic")
    public KStream<?, AggregatedCountry> process(KStream<Object, Country> input) {
        return input
                .groupBy((key, value) -> value.getCountryCode())
                .aggregate(this::initialize,
                        this::aggregateAmount,
                        materializedAsPersistentStore("countries", Serdes.String(),
                                Serdes.serdeFrom(new JsonSerializer<>(),
                                        new JsonDeserializer<>(AggregatedCountry.class))))
                .toStream()
                .map((key, value) -> new KeyValue<>(null, value));
    }
    @StreamListener("countries")
    @SendTo("daily-statistic")
    public KStream<?, List<DailyStatistics>> daily(KStream<Object, Country> input) {
        return input
                .groupBy((key, value) -> value.getCountryCode())
                .aggregate(this::initializeDailyStatistics,
                        this::dailyStatistics,
                        materializedAsPersistentStore("daily", Serdes.String(),
                                Serdes.serdeFrom(new JsonSerializer<>(),
                                        new JsonDeserializer<>(List.class))))
                .toStream()
                .map((key, value) -> new KeyValue<>(null, value));
    }

But when I start Spring Boot app I get this error.

Exception in thread "kafka-stream-f4f8166b-cbeb-42ca-b461-2b3a23885a5d-StreamThread-1" java.lang.IllegalStateException: Consumer was assigned partitions [kafka-stream-daily-repartition-0] which didn't correspond to subscription request [kafka-stream-countries-repartition, countries]
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.handleAssignmentMismatch(ConsumerCoordinator.java:218)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:264)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:424)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:358)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:353)
    at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:963)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:863)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:819)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:788)

I guess I need separate application id for each StreamListener methods but how can I configure it in application.yml file if I am listening to the same topic ?

score 2 · Accepted Answer · answered Jun 19 '20 at 20:10

You need to provide two separate input bindings (and both of those can point to the same topic). You cannot use the same binding name on multiple StreamListeners. You can then set application.id for multiple StreamListener based processors on the input bindings. For e.g.

spring.cloud.stream.kafka.streams.bindings.countries1.consumer.applicationId

and

spring.cloud.stream.kafka.streams.bindings.countries2.consumer.applicationId

See this section from the ref docs.

Thanks for such quick response. Your solution worked perfectly well. — Mykhailo Yablon, Jun 19 '20 at 20:18

score 2 · Answer 2 · answered Oct 24 '20 at 06:02

You are reading "countries" topic twice, would be better if you read once from "countries", and send data to "daily-statistic" and "aggregated-statistic".

Reading twice is not the same thing as concurrent processing. If you want concurrency configure this parameter:


spring:
  cloud.stream:
    bindings:
      countries:
        destination: countries-topic
        consumer.concurrency: 6

And you can use a topology like:


@StreamListener("countries")
@SendTo({"daily-statistic", "aggregated-statistic"})

Multiple StreamListeners to same Topic with Spring Cloud Stream connected to Kafka

2 Answers2