Stack Overflow
Stack Overflow

Questions tagged [amazon-kcl]

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

Source: https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html

99 questions
0
votes
1 answer

AWS Kinesis KCL skips records added before startup

I started to use both KPL and KCL to exchange data between services. But whenever consumer service is offline, all data sent by KPL are lost forever. So I get only those chunks of data that were sent while consumer service is up and its…
Alesto
  • 639
  • 4
  • 13
  • 29
0
votes
1 answer

How to consume records from Kinesis Data Streams using KCL

I am trying to consume Kinesis data streams using Kinesis Client Library similar to https://github.com/aws-samples/amazon-kinesis-learning. But in this example they scheduled the process. I want to consume records afferent without scheduler. I don't…
Achaius
  • 5,904
  • 21
  • 65
  • 122
0
votes
1 answer

AWS Kinesis Enhanced fan-out Java example

I have an application that consumes records from Kinesis streams and processes them further but the performance is quite low, So now I am planning to migrate to the Kinesis Enhanced fan-out consumer using KCL 2.x to improve its performance. As the…
Random Coder
  • 13
  • 6
0
votes
1 answer

KCL stops processing data after throwing error Cancelling subscription, and restarting

KCL ShardConsumerSubscriber:131 - shardId-000000000000: Last request was dispatched at 2020-04-28T12:57:25.166Z, but no response as of 2020-04-28T12:58:00.435Z (PT35.269S). Cancelling subscription, and restarting." But never restarts application…
Vivek Basidoni
  • 88
  • 2
  • 10
0
votes
0 answers

Flink kinesis connector - KCL

We are using the kinesis flink connector to consume and produce records into kinesis from flink. Since it is using KCL, it should make entries in dynamoDB with the offset for the kinesis streams it consumes. We are not able to see any tables with…
justlikethat
  • 329
  • 2
  • 12
0
votes
0 answers

Error while running KCL consumer with the localstack Kinesis

Caught exception while syncing Kinesis shards and leases Caused by: io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 3c21444f43
khitsik
  • 3
  • 2
0
votes
1 answer

How can I access a specific record in a Kinesis stream?

I have a Kinesis stream active and running, to which I wrote a producer and a consumer using KCL library. Is there any possible way to access a specific record, with the use of sequence number and partition key?
Nadav Miran
  • 59
  • 1
  • 5
0
votes
0 answers

How to implement a worker thread that will process Kinesis records and update GUI in javaFx?

I'm working on a micro-services monitoring app. My app supposes to update a GUI accordingly when receiving a new consumed record, meaning: When I receive a new record: 1)I check if the request it represents is a part of a legal flow, and if that…
Elon
  • 11
  • 1
0
votes
1 answer

How to pause KCL client in Java

We have a Spring application using the Java KCL client to consume records off of an Kinesis stream. Is there a way we could somehow 'pause' the KCL to stop processing records and then restart it later safely? What would be the recommended way to go…
John Doe
  • 205
  • 6
  • 17
0
votes
1 answer

How Can I See My Shard ID for KCL in My AWS Console?

I'm new to AWS Kinesis. I'm trying to work with KCL version 1 (following the amazon instructions here: https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-implementation-app-java.html#kcl-java-interface-original) I want to…
gusfring
  • 35
  • 6
0
votes
1 answer

Process multiple Kinesis streams within single Java process

I would like to process multiple Kinesis streams using KCL within the same Java process. The idea is simple: make a new KCL instance for each stream and then run the workers concurrently. My question is whether in this case all KCL instances are…
Mark
  • 253
  • 1
  • 5
  • 22
0
votes
1 answer

Is it possbile to read Kinesis stream on demand with no lag

I have following use case I have kinesis stream having user data. I want to read kinesis stream based on user action. Filter records based on user input, keep filtering for some time period, let say 5 minutes. Keep returning these filtered batches…
Amit Kumar
  • 825
  • 1
  • 9
  • 19
0
votes
1 answer

Is there a way to read from Dynamo DB stream with a fixed no of workers and leases without any issue

I am continuously publishing data into dynamoDB which has stream enabled. I am reading this stream using DynamoDB apadter of KCL. I am using 1 KCL worker with 5 leases. At the time of creation my Dynamo table had 1 partition(1 RCU and 999WCU). When…
user2677679
  • 259
  • 2
  • 12
0
votes
0 answers

How KCL internally spawns new RecordProcessor

I have a java consumer (KCL implementation) to consume messages from 1 Kinesis stream with 2 shards. The KCL creates 2 RecordProcessors correlates to each shard. Then I put my machine to sleep, so JVM suspended. When wake up, the KCL library…
ethan
  • 1,881
  • 2
  • 17
  • 31
0
votes
1 answer

KCL how to know the number of records for one ProcessRecordsInput?

I have a Kinesis consumer written in Java with KCL. From the consumer side, I can throttle the rate with setMaxRecords method to set the maximum limit of how many records within one batch. My question is: if I don't set any limit, what is the logic…
ethan
  • 1,881
  • 2
  • 17
  • 31
1 2 3 4 5
6
7