Dynamic kinesis checkpoint with DynamoDbMetaDataStore

Question

I have following configuration for checkpoint store:

public DynamoDbMetaDataStore dynamoDbMetaDataStore() {
    String url = consumerClientProperties.getDynamoDB().getUrl();
    final AmazonDynamoDBAsync amazonDynamoDB = AmazonDynamoDBAsyncClientBuilder.standard()
        .withEndpointConfiguration(new EndpointConfiguration(
            url,
            Regions.fromName(awsRegion).getName()))
        .withClientConfiguration(new ClientConfiguration()
            .withMaxErrorRetry(consumerClientProperties.getDynamoDB().getRetries())
            .withConnectionTimeout(consumerClientProperties.getDynamoDB().getConnectionTimeout())).build();
    DynamoDbMetaDataStore dynamoDbMetaDataStore = new DynamoDbMetaDataStore(amazonDynamoDB, "consumer-client-library");
    return dynamoDbMetaDataStore;
  }

  public KinesisMessageDrivenChannelAdapter kinesisInboundChannelChannel(
      AmazonKinesis amazonKinesis, String[] streamNames) {
    KinesisMessageDrivenChannelAdapter adapter =
        new KinesisMessageDrivenChannelAdapter(amazonKinesis, streamNames);
    adapter.setConverter(null);
    adapter.setOutputChannel(kinesisReceiveChannel());
    adapter.setCheckpointStore(dynamoDbMetaDataStore());
    adapter.setCheckpointMode(CheckpointMode.manual);
    adapter.setListenerMode(ListenerMode.batch);
    return adapter;
  }

When consumer starts it is looking for checkpoint values by creating following key

private String buildCheckpointKeyForShard(String stream, String shardId) {
    return this.consumerGroup + ":" + stream + ":" + shardId;
  }

This means that I should have the keys created before consumer starts, which requires to get all shards from kinesis.

Why there is no generic way of creating keys if they are absent? why checkpoint store does not use putIfAbsent so the keys could be generated dynamically?

Thank you

Dynamic kinesis checkpoint with DynamoDbMetaDataStore

0 Answers0