KCL stops processing data after throwing error Cancelling subscription, and restarting

Question

KCL

ShardConsumerSubscriber:131 - shardId-000000000000: Last request was dispatched at 2020-04-28T12:57:25.166Z, but no response as of 2020-04-28T12:58:00.435Z (PT35.269S). Cancelling subscription, and restarting." But never restarts application and no data is processed after that. Maven dependency used

<dependency>
            <groupId>software.amazon.kinesis</groupId>
            <artifactId>amazon-kinesis-client</artifactId>
            <version>2.2.2</version>
</dependency>

And the Kinesis configuration

KinesisAsyncClient kinesisClient = KinesisAsyncClient.builder()
                .credentialsProvider(new MyCredentialProvider(configVals)).region(region).build();
InitialPositionInStreamExtended initialPositionInStreamExtended = InitialPositionInStreamExtended
        .newInitialPosition(InitialPositionInStream.TRIM_HORIZON);
RetrievalConfig retrievalConfig = configsBuilder.retrievalConfig()
        .retrievalSpecificConfig(new PollingConfig(configVals.getStreamName(), kinesisClient)
                .idleTimeBetweenReadsInMillis(10000).maxRecords(50).kinesisRequestTimeout(Duration.ofSeconds(100)));
retrievalConfig.initialPositionInStreamExtended(initialPositionInStreamExtended);

score 0 · Answer 1 · answered Oct 05 '20 at 13:58

As it's stated in issue KCL 2.0 stops consuming from some shards #463, the main reason can be the fact that the processor is not able to process the input in 35 seconds. This value is hardcoded in ShardConsumer in MAX_TIME_BETWEEN_REQUEST_RESPONSE variable and is increased to 1 minute in later versions (I'm using kcl version 2.2.10 and it's 60 seconds already).

I've also experienced this issue and additionally had to override kinesis client http client settings to setup higher timeouts - something like that way:

val kinesisClient = KinesisClientUtil
      .adjustKinesisClientBuilder(
        KinesisAsyncClient.builder.credentialsProvider(credentialsProvider).region(awsRegion)
      )
      .httpClientBuilder(
        NettyNioAsyncHttpClient.builder
          .maxConcurrency(Integer.MAX_VALUE)
          .maxPendingConnectionAcquires(kinesisHttpMaxPendingConnectionAcquires)
          .maxHttp2Streams(kinesisHttpMaxConnections)
          .connectionAcquisitionTimeout(kinesisHttpConnectionAcquisitionTimeout)
          .connectionTimeout(kinesisHttpConnectionTimeout)
          .readTimeout(kinesisHttpReadTimeout)
          .http2Configuration(
            Http2Configuration.builder
              .initialWindowSize(10 * 1024 * 1024)
              .healthCheckPingPeriod(kinesisHttpHealthCheckPingPeriod)
              .maxStreams(kinesisHttpMaxConnections.longValue())
              .build
          )
          .protocol(Protocol.HTTP2)
      )
      .build()

I've set all the timeouts to 120 seconds and it did the trick.

KCL stops processing data after throwing error Cancelling subscription, and restarting

1 Answers1