Spring Integration Kinesis adapter and consumer groups

Question

I have a kinesis consumer application developed using spring-integration-aws version 1.1.0.RELEASE.

In my tests, I am running two instance of this application in the same consumer group and consuming from a stream with two shards. In my tests I realized KinesisMessageDrivenChannelAdapter would distribute messages in three ways:

1. All messages delivered to one consumer
2. messages distributed to the both consumers (not evenly)
3. Both consumers received same messages

From producer side, messages are distributed evenly between two shards. I would like to know how kinesis adapter distributes messages among consumers and if supported how I can get an even distribution among consumers.

Thank you

UPDATE (Adapter Configuration)

@Bean
  public KinesisMessageDrivenChannelAdapter kinesisInboundChannelAdapter(
      AmazonKinesis amazonKinesis) {
    String[] streamNames = this.consumerClientProperties.getKinesis().getStreamNames();
    KinesisMessageDrivenChannelAdapter adapter =
        new KinesisMessageDrivenChannelAdapter(amazonKinesis, streamNames);
    adapter.setConverter(null);
    adapter.setOutputChannel(new QueueChannel());
    adapter.setCheckpointStore(dynamoDbMetaDataStore());
    adapter.setCheckpointMode(CheckpointMode.record);
    adapter.setStartTimeout(10000);
    adapter.setConsumerGroup(consumerClientProperties.getName());
    adapter.setListenerMode(ListenerMode.record);
    adapter.setDescribeStreamRetries(1);
    return adapter;
  }

  @Bean
  public DynamoDbMetadataStore dynamoDbMetaDataStore() {
    DynamoDbMetadataStore dynamoDbMetaDataStore = new DynamoDbMetadataStore(amazonDynamoDB(),
        consumerClientProperties.getName());
    return dynamoDbMetaDataStore;
  }

Answer 1

It is recommended to everyone to upgrade to the latest Spring Integration AWS 2.0: https://spring.io/blog/2018/08/21/spring-integration-for-aws-2-0-ga-and-spring-cloud-stream-kinesis-binder-1-0-ga

There were a plenty of fixes done on the Kinesis consumer level and now we have there a leader election do not subscribe to the same shard more than once.

The idea is to have strict ordering when we process records, therefore only one thread per cluster should have access to one shard. That thread may process several shards though.

Anyway if you use two instance of the application you need to inject a MetadataStore with shared data based, e.g. DynamoDbMetadataStore.