2

We are using kinesis as our queue to store the events triggering lambda functions on AWS. We intermittently run into ProvisionedThroughputExceededException, though our payloads are really small. After rounds of discussion with AWS architects, they told us that if there were more than 5 lambdas listening to the same stream, high chances to encounter this exception. Anyone is experiencing the same issue? Would love to see how u handle it…

Hammer
  • 8,538
  • 12
  • 44
  • 75
  • 3
    Is this during reads or writes? Yes, you can have only 5 reads in 1000ms, meaning if you have happen to have 10 consumers then you are making potentially 10 read requests within 1000ms and it will blow up. – prayagupa Jun 16 '17 at 08:42
  • Ya, it is for read. Any suggestions on how to handle it? Create a fan-out pattern using SNS+SQS? – Hammer Jun 16 '17 at 09:02
  • 1
    Yes, I have seen people using fanout [for this particular problem](https://github.com/awslabs/aws-lambda-fanout). You can fanout to another stream. I won't add SNS or SQS, they have their own problems again like event size, writes etc. This might be helpful too - [How to fanout an AWS kinesis stream?](https://stackoverflow.com/a/40066453/432903) . You might need to handle error scenarios in case you were not able to write to **target stream**, you are basically replicating the same source of truth. – prayagupa Jun 16 '17 at 09:14
  • Thanks. Or in the first place, should not use Kinesis as an event store? – Hammer Jun 16 '17 at 09:33
  • 1
    Very true, if its already not in production, you can find the eventstore options. Kafka could be option if you have enough resources. I am stuck to Kinesis because of resources, but been working on [Stream-Driver](https://github.com/duwamish-os/eventstream-skipper#stream-api) so that once I get my Kafka deployment ready can switch from kinesis to Kafka. I had done [Stream-Comparison](https://github.com/duwamish-os/eventstream-skipper/blob/master/StreamComparison.md) while choosing the stream, which might be helpful in case you have not looked into Kafka. – prayagupa Jun 16 '17 at 09:41

0 Answers0