0

We have a spring cloud stream app consuming from Kinesis stream with a single shard. The application has a consumer.concurrency of 2 and the spring.cloud.stream.instanceIndex and spring.cloud.stream.instanceCount are correctly set on each instance.

We have also added spring.cloud.stream.bindings.binding-target.group setting but this had no effect.

We have a statefulset with a minimum of 2 replicas.

The problem is that, when the messages are consumed from Kinesis, both instances are consuming the same data.

If we increase the number of shards from 1 to 2 then this problem goes away.

We are using the following maven dependencies:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-stream-binder-kinesis</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-dependencies</artifactId>
    <version>2021.0.2</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>

Why are the services consuming the same data when there is a single shard?

Dan Nemes
  • 300
  • 3
  • 16
  • Any chances to have some simple project from you to reproduce, please? – Artem Bilan Jun 08 '22 at 15:52
  • It is not possible to share the project. – Dan Nemes Jun 08 '22 at 19:40
  • No problem! Just need to understand what is your environment: how many instances of your app, what are your configuration properties, what is the version of this Kinesis Binder? I mean from the code perspective what you say is not possible since there is lock set on a consumed shard and another one cannot do anything if both of them are in the same group. – Artem Bilan Jun 08 '22 at 19:43
  • We are using the following maven artifactId versions: spring-cloud-stream-kinesis-binder version 2.2.0 spring-cloud-dependencies version 2021.0.2 (this comes with spring-cloud-stream version 3.2.3) What other configuration/versions would you need to know? We have a service deployed in K8s as statefulset with a minimum of 2 replicas. – Dan Nemes Jun 09 '22 at 06:46
  • What you are saying just does not make any sense if group is really the same throughout your instances. Even if we assign the same shard to several instances, we still set a lock for its name including the provided consumer group. – Artem Bilan Jun 09 '22 at 19:41
  • We are still looking into this. We are definitely having this issue and this is a service that will be going in prod soon. We will try to use the latest versions to see if this fixes our problem. – Dan Nemes Jun 10 '22 at 05:35

0 Answers0