How do we process the kinesis stream data every one hour?

Question

I have a kinesis stream which is being continuously written with putRecord. In the consumer I consume using processRecords of KCL. This process records gets called whenever there is a record in the stream.I need this process records to be excuted every X hour.

Here are the things that I tried.

kinesisClientLibConfiguration = new KinesisClientLibConfiguration(applicationName, kinesisStreamName,
            credsProvider, workerId).withInitialPositionInStream(initialPositionInStream)
                    .withCallProcessRecordsEvenForEmptyRecordList(true)
                    .withIdleTimeBetweenReadsInMillis(3600000) //1 hr in millis
                    .withKinesisEndpoint(kinesisEndpoint);

This does'nt seem to be working. It throws the below exception

    [INFO] 2018-02-08T18:41:24.124 com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask getRecordsResult  ShardId shardId-000000000003: getRecords threw ExpiredIteratorException - restarting after greatest seqNum passed to customer
com.amazonaws.services.kinesis.model.ExpiredIteratorException: Iterator expired. The iterator was created at time Thu Feb 08 13:06:04 UTC 2018 while right now it is Thu Feb 08 13:11:23 UTC 2018 which is further in the future than the tolerated delay of 300000 milliseconds. (Service: AmazonKinesis; Status Code: 400; Error Code: ExpiredIteratorException; Request ID: d72ba81b-9a14-cf1d-85dd-e6a01179f91c)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2276)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2252)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.executeGetRecords(AmazonKinesisClient.java:1062)
    at com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:1038)
    at com.amazonaws.services.kinesis.clientlibrary.proxies.KinesisProxy.get(KinesisProxy.java:158)
    at com.amazonaws.services.kinesis.clientlibrary.proxies.MetricsCollectingKinesisProxyDecorator.get(MetricsCollectingKinesisProxyDecorator.java:74)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisDataFetcher.getRecords(KinesisDataFetcher.java:74)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.SynchronousGetRecordsRetrievalStrategy.getRecords(SynchronousGetRecordsRetrievalStrategy.java:31)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.BlockingGetRecordsCache.getNextResult(BlockingGetRecordsCache.java:50)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.getRecordsResultAndRecordMillisBehindLatest(ProcessTask.java:377)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.getRecordsResult(ProcessTask.java:342)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:159)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:49)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:24)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

Can someone please let me know if there is a work around for achieving this? Thanks

This is arguably a bug in the KCL: it should either reject a configuration that exceeds the shard iterator timeout of 5 minutes (300 seconds) or catch the exception and reload the iterator. You could look through the [issues](https://github.com/awslabs/amazon-kinesis-client/issues) for the library on GitHub, to see if anyone else has reported the problem. If not, I recommend adding a new issue. — kdgregory, Apr 06 '18 at 12:46
Your error looks like the shardIterator has just barely expired (just over 5 minutes) - I'm not sure how withIdleTimeBetweenReadsInMillis of 1 hour would relate to this - is this really the error message from your posted configuration? — Krease, Jun 13 '18 at 17:57

How do we process the kinesis stream data every one hour?

0 Answers0