Records are not consumed when adding checkpointer

Question

I have following configuration for KinesisMessageDrivenChannelAdapter, when I remove dynamoDbMetaDataStore as checkpointer, messages are received correctly, but when I add it back records are always empty. I debugged the code and KinesisMessageDrivenChannelAdapter.processTask() line 776 (version 2.0.0.M2) returns empty records.

UPDATE:

public DynamoDbMetaDataStore dynamoDbMetaDataStore() {
    String url = consumerClientProperties.getDynamoDB().getUrl();
    final AmazonDynamoDBAsync amazonDynamoDB = AmazonDynamoDBAsyncClientBuilder.standard()
        .withEndpointConfiguration(new EndpointConfiguration(
            url,
            Regions.fromName(awsRegion).getName()))
        .withClientConfiguration(new ClientConfiguration()
            .withMaxErrorRetry(consumerClientProperties.getDynamoDB().getRetries())
            .withConnectionTimeout(consumerClientProperties.getDynamoDB().getConnectionTimeout())).build();
    DynamoDbMetaDataStore dynamoDbMetaDataStore = new DynamoDbMetaDataStore(amazonDynamoDB, "consumer-test");
    return dynamoDbMetaDataStore;
  }

  public KinesisMessageDrivenChannelAdapter kinesisInboundChannel(
      AmazonKinesis amazonKinesis, String[] streamNames) {
    KinesisMessageDrivenChannelAdapter adapter =
        new KinesisMessageDrivenChannelAdapter(amazonKinesis, streamNames);
    adapter.setConverter(null);
    adapter.setOutputChannel(kinesisReceiveChannel());
    adapter.setCheckpointStore(dynamoDbMetaDataStore());
    adapter.setConsumerGroup(consumerClientProperties.getName());
    adapter.setCheckpointMode(CheckpointMode.manual);
    adapter.setListenerMode(ListenerMode.record);
    adapter.setStartTimeout(10000);
    adapter.setDescribeStreamRetries(1);
    adapter.setConcurrency(10);
    return adapter;
  }

Thank you

And where do they go? How about some DEBUG logs to investigate what and how happens in your application? Maybe you even can come up with the simple project on GitHub to let us to play with it. But we definitely need a good instruction how to reproduce. The code you show now is fully not enough. You might just have a bad credentials to connect to AWS, so that's how there is no any consumed events at all. And so on, and so on... Let's just don't play spy games here ! — Artem Bilan, Jun 25 '18 at 16:19
You are totally right! The reason I don't receive any messages is related to adding `dynamoDbMetaDataStore`. Without setting checkpointer message consumption works fine, but when I add checkpointer then in `KinesisMessageDrivenChannelAdapter.processTask()` all shards are empty of records. I really don't know where should I look into. I updated my post with dynamoDbMetaDataStore configuration. — sansari, Jun 25 '18 at 18:54

score 0 · Accepted Answer · answered Jun 25 '18 at 19:12

0

I recommend you to test your solution with the latest 2.0.0.BUILD-SNAPSHOT.

There is already an option like:

/**
 * Specify a {@link LockRegistry} for an exclusive access to provided streams.
 * This is not used when shards-based configuration is provided.
 * @param lockRegistry the {@link LockRegistry} to use.
 * @since 2.0
 */
public void setLockRegistry(LockRegistry lockRegistry) {

where you would need to inject a DynamoDbLockRegistry for better checkpoint management.

For that purpose you would also need to add this dependency:

compile("com.amazonaws:dynamodb-lock-client:1.0.0")

There indeed might be some issues with filtering in that M2 yet...

answered Jun 25 '18 at 19:12

Artem Bilan

113,505
11
91
118

I got `2.0.0.BUILD-SNAPSHOT` but there is no `setLockRegistry` in `KinesisMessageDrivenChannelAdapter` – sansari Jun 25 '18 at 20:03
Well, that’s impossible: https://github.com/spring-projects/spring-integration-aws/commit/ebd562d6da23c305ae8d4608e04712dedf97cfe9. And this stuff is added to the Kinesis Binder already – Artem Bilan Jun 25 '18 at 20:06
After changing to snapshot version I am getting this error: `java.lang.NoClassDefFoundError: org/springframework/cloud/aws/core/io/s3/SimpleStorageResourceLoader` – sansari Jun 25 '18 at 20:53
version `2.0.0.BUILD-SNAPSHOT` brings in spring-cloud-version `2.0.0.RELEASE` which does not have `SimpleStorageResourceLoader` in specified package in previous comment – sansari Jun 25 '18 at 21:23
Well, that has been removed anyway: https://github.com/spring-cloud/spring-cloud-aws/commit/0022146e87103aedf592fee65ad2aec64624ca81. Upgrading to Spring Cloud 2.0 you definitely don't have choice unless revise that `SimpleStorageResourceLoader` usage. You just don't need to use it any more! – Artem Bilan Jun 25 '18 at 21:54
I created a table name `lockTable` with partitionKey `lockKey` and sort key `sortKey`. `KinesisMessageDrivenChannelAdapter.tryLock(key)` line 631 always times out. – sansari Jun 27 '18 at 15:02
Right, we are working on that in the latest commits. See discussion here: https://github.com/spring-projects/spring-integration-aws/issues/90 – Artem Bilan Jun 27 '18 at 15:07
According to your last comment on github link, I removed LockRegistry, but again I go back the issue no record is received in `processTask` line 953. As I said before it works fine without `dynamoDbMetaDataStore` – sansari Jun 27 '18 at 18:33
Is it going to be hard for you to share with me somewhere on GiutHub some simple project to investigate a situation? – Artem Bilan Jun 27 '18 at 18:56
1

Not at all, I will send you a link to github. – sansari Jun 27 '18 at 19:17
Artem, here is the project https://github.com/saeedansari/consumer-library configuration is in `com.loyalty.consumer.configuration.ConsumerClientConfiguration` I appreciate your help – sansari Jun 27 '18 at 20:07
How does it work if you don't use a ` adapter.setCheckpointMode(CheckpointMode.manual);` , but some other available? – Artem Bilan Jun 27 '18 at 20:42
I changed it to the batch mode, and did step by step debugging. Debugging took a while and then a record was received. I don't think it is because of changing to batch but maybe a happen-before issue? Running has the same result: no record. – sansari Jun 27 '18 at 21:26
There is a log like this `logger.trace("Processing records: " + records + " for [" + ShardConsumer.this + "]");`, so turn on the TRACE for the `org.springframework.integration.aws.inbound.kinesis` category and let's see what's going on! I also would appreciate if you share some instruction for your application how to reproduce. – Artem Bilan Jun 28 '18 at 01:02
1

So, what I found in your project that you don't mark with the `@Bean` a `amazonDynamoDB()` and `dynamoDbMetaDataStore()`. The problem with the `DynamoDbMetaDataStore` that its `afterPropertiesSet()` is not called and therefore the table is not created. – Artem Bilan Jun 28 '18 at 14:55
1

Then I changed to this `adapter.setCheckpointMode(CheckpointMode.record);` and this `adapter.setListenerMode(ListenerMode.record);` and test has started to pass. – Artem Bilan Jun 28 '18 at 14:58
Cool! Thanks a lot Artem. I created the table manually with configs `CheckpointMode.batch` and `ListenerMode.batch` and it started to work. But definitely I will fix the bug you found on my project. I really appreciate your help. – sansari Jun 28 '18 at 15:12
Right, with the `batch` mode you need to expect a `List` on the consumer side, when you don't configure `converter` - ` adapter.setConverter(null);` – Artem Bilan Jun 28 '18 at 15:17
1

changed to record and marked `amazonDynamoDB` and `DynamoDbMetaDataStore` as `@Bean`. perfectly works! – sansari Jun 28 '18 at 15:46

Records are not consumed when adding checkpointer

1 Answers1