0

I have following use case

  • I have kinesis stream having user data.
  • I want to read kinesis stream based on user action.
  • Filter records based on user input, keep filtering for some time period, let say 5 minutes.
  • Keep returning these filtered batches to user for 5 minutes.
  • After timeout stop reading kinesis

Question: Is there a way of reading kinesis on demand, without any lag using KCL or any other library. Ley say i can have KCL jvm apps setup which is not reading currently, whenever it gets user action, just start reading. Similarly stop reading after some timeout or further user action.

I can write a logic which can do that but would like to know if there is anything built in KCL.

Amit Kumar
  • 825
  • 1
  • 9
  • 19
  • what exactly do you mean by "without any lag using KCL"? There's going to be some small lag between event generation and event consumption, since the data has to be transmitted to and from AWS for storage in the stream. – Krease Oct 24 '18 at 21:50
  • Yup that lag is acceptable. – Amit Kumar Oct 25 '18 at 07:37

1 Answers1

0

KCL does this for you - as long as the KCL application is running, it will continually read from the stream. Doesn't matter if most of the time your stream returns no data - it'll just keep running and doing nothing until there is data, at which point your processing code kicks in.

You can configure the timeout it takes when there are no records with KinesisClientLibConfiguration.idleTimeBetweenReadsInMillis - the default is 1 second. There are a lot of configuration options here to fine tune the behavior as needed.

Now, if your streams will be empty a lot of the time, it may be more cost effective to use AWS Lambda to process your streams, which will process your records on-demand without needing to run (and pay for) hardware to continually do read operations from the stream.

Krease
  • 15,805
  • 8
  • 54
  • 86