Amazon kinesis KCL 1.14.3 - multiple shards and instances issue

Asked Jun 21 '21 at 19:13

Active Jun 22 '21 at 05:49

Viewed 207 times

I have a two instances of my app and two shards in kinesis.

Scenario,

Launching instance 1. Application gets two shards. Everything is ok.
Launching instance 2 after a few time laters. When Instance 1 processing records from both shards then in the background I have information about shutdown on shard2.

Invoking shutdown() for shard shardId-0000000000XX, concurrencyToken: 07b1aba8-0aa3-48c4-9992-XXXXXXXX, original Shutdown reason: ZOMBIE. childShards:null

I make checkpoint after finishing process records and now I loose lease for second shard then I can't make checkpoint and instance 2 received new records from kinesis immediately.

I try block method shutdown in instance1 to wait to finish records processing but instance2 received records from kinesis at the same time.

How to override any method to keep lease for shard 2 and gracefully finish processing records?

edited Jun 22 '21 at 05:49

asked Jun 21 '21 at 19:13

Mateusz

Did you find a solution for this problem? Getting the same issue. – Jonathan Apr 20 '22 at 13:19
Unfortunately not, is not something I can't live with. If you find solution please give a feedback :) – Mateusz Apr 21 '22 at 05:57

Amazon kinesis KCL 1.14.3 - multiple shards and instances issue

0 Answers0