1

Suppose i have a single shard Kinesis stream which allows 1MBps incoming. I have a producer using KPL which is trying to pump 2MBps (using async call of KPL).

Given the KPL behaviour of retrying, i would assume it would buffer events for some time to retry and at some point it would either exhaust its memory or start dropping events.

Which of the above case would happen and if it starts dropping event, how can i monitor it? I am unable to find any dropped events metric in Cloudwatch.

---- Updated below with a graph ----- enter image description here

dy10
  • 117
  • 1
  • 8

1 Answers1

0

you can use WriteProvisionedThroughputExceeded metrics to get the count of events dropped in a specific period as mentioned here

Vinujan.S
  • 1,199
  • 14
  • 27
  • it is not clear from the docs if this number comes after retry attempts or before retry. As per my understanding, KPL will retry if threshold is exceeded. Updating the question to include a graph of retries and throughputExceeded. – dy10 Apr 18 '18 at 07:25
  • I don't think KPL related stat and Stream related stats are strongly related. My assumption is Stream related stats are collected solely based on input coming from multiple producers in kinesis stream end. KPL send stats to cloud watch from its application point of view. My assumption is that number is not related to retry attempts. – Vinujan.S Apr 18 '18 at 12:39
  • @dy10 isn't above is convincing? – Vinujan.S Apr 19 '18 at 12:25
  • I appreciate your response, but this is based on assumptions which I don't find convincing. WriteProvisionedThroughputExceeded could as well just means that for than second, input was more than expected, but then KPL retries and it goes thru in next second and nothing is dropped (because there was no incoming data in next second, so KPL could send previous data). – dy10 Apr 20 '18 at 13:28