1

In AWS Kinesis Streams, what does this limit exactly mean:

GetShardIterator has a limit of 5 transactions per second per account per open shard.

In particular, what is an "open shard"? A shard that has a shard iterator that is not expired yet? Is this to encourage using open iterators and discourage from opening too many iterators for the same shard at once?

Or does it mean that up to 5 GetRecords transactions can be sent to the same shard iterator per second?

Behrang
  • 46,888
  • 25
  • 118
  • 160

1 Answers1

1

  1. A shard can be explained as a virtual queue, that holds data in the steam. There can be multiple shards in a stream and when 1 record is pushed to the stream (via PutRecord API call) it is placed on a shard based on that record's partition key.

  2. An open shard is a shard that can accept new records. If you merge 2 shards then they become closed. Likewise when you split a shard, it becomes closed. Sample explanation on merge/split here: How to decide total number of partition keys in AWS kinesis stream?

  3. Shard iterator is a little bit different, it is the polling rate of a shard in the consumer application. Most of the time your application will not bother how a shard iterator works (KCL: amazon-kinesis-client will handle it internally). The "limit of 5 transactions per second per account per open shard" means the interval of polling to a shard is limited, that is explained here: Kinesis max shard reads/sec and multiple consumers

az3
  • 3,571
  • 31
  • 31