0

For example, RabbitMQ has a way in setting queue limits. If that limit is reached the new messages from publishers will be rejected, thus applying some kind of backpressure that starts from consumers to the producers. (since messages in queues means not processed by consumers).

Is there a way to assure this kind of behavior for brokers like Kinesis in which the consumers are allowed to pull messages and not the broker pushes to them, like RabbitMQ.

In case of Kinesis, similar to Kafka, the state of the consumers, offset of consumption and so on, is kept in a different entity, DynamoDB for Kinesis and I know this can be trickier to have something like unprocessed records limits out of the box.

Does anyone know if there is some settings you can use, maybe by the use of KCL / KPL client library, or something ?

raduone
  • 191
  • 1
  • 10

1 Answers1

1

No. AWS Kinesis does not provide the feature you want unfortunately. There is no way to stop producer writing into a Kinesis stream if the consumer cannot catch up in processing.

In fact this is one of the advantage of using Kinesis, it allows unlimited buffering of data up to the configured retention time for free. The only time it provides back pressure is when the producer writes too much data too fast because of the Amazon Kinesis API limit: https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html

If you want a limited size "queue", maybe you want to look into AWS SQS where it has a lower limit of 12000 inflight messages?

If you do want to use Kinesis, you might want to build a custom solution to feed the consumer delay back to the producer. For example, implement custom logic in the producer to monitor the consumer delay ('MillisBehindLatest') using AWS Cloudwatch (See https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html) and stop when the consumer is falling behind.

Jackyjjc
  • 552
  • 1
  • 3
  • 13