Stack Overflow
Stack Overflow

Questions tagged [amazon-kcl]

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

Source: https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html

99 questions
0
votes
0 answers

KCL ShardSyncTask - Caught exception while sync'ing Kinesis shards and leases

How serious is this exception coming from kcl library? I am using Java Amazon Kinesis Adaptor(1.5.4) on DynamoDB Streams. Below is the stack trace for the error I received: ERROR c.a.s.k.c.l.w.ShardSyncTask - Caught exception while sync'ing Kinesis…
shashijais789
  • 19
  • 4
0
votes
0 answers

How to configure KCL worker properties for Amazon KCL client

We have a java based KCL worker consuming records from DynamoDB streams. I want to understand based on high number of partition in table and shards in streams, what values should be given while creating worker. Also these tables are actively merging…
Mahesh Nagarwal
  • 55
  • 1
  • 1
  • 6
0
votes
1 answer

KCL doesn't PUT or GET and items from DynamoDB checkpoint or locks tables

I implemented reading messages from Kinesis using KCL, but when i check DynamoDB tables, i found that only group table have the list of shards, but checkpoint table and locks table doesn't have any items: Any idea why KCL doesn't PUT or GET items…
mibrahim.iti
  • 1,928
  • 5
  • 22
  • 50
0
votes
0 answers

KCL Encountered an exception while renewing a lease

I implement reading from AWS/Kinesis stream using KCL, but after read many messages from shards, suddenly i see this error in server logs: ERROR 1 --- [oordinator-0000] c.a.s.kinesis.leases.impl.LeaseRenewer : Encountered an exception while…
mibrahim.iti
  • 1,928
  • 5
  • 22
  • 50
0
votes
0 answers

KCL reader AWS instances goes down quickly due to heap memory exception

I wrote next implementation to read messages from AWS Kinesis and deployed it on AWS ECS: Application.java import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import…
mibrahim.iti
  • 1,928
  • 5
  • 22
  • 50
0
votes
1 answer

Reprocessing messages AWS kinesis data stream message using KCL consumer

A KCL consumer is running using Auto Scaling Group(ASG) configured according to the number of provisioned shards of the kinesis data stream on EC2 machines which means if the kinesis data stream has n number of provisioned shards then the maximum n…
Shubhank Gupta
  • 705
  • 2
  • 10
  • 27
0
votes
1 answer

Kinesis Stream Consumption: LATEST v/s TRIM_HORIZON

I have a use case where I need to keep the Kinesis trigger on my consumer (let's call it Lambda B) be disabled, while the producer (let's call it Lambda A) writes to the Kinesis stream. Once the write is complete, I intend to enable the trigger, and…
Debapratim Chakraborty
  • 375
  • 3
  • 15
0
votes
2 answers

k8s pod keeps crashing (CrashLoopBackOff), but describe shows "Completed" with code 0?

I'm running a node application built on top of AWS's java KCL lib on k8s. Every 5 minutes or so the container crashe with "CrashLoopBackOff" and restarts - I can't figure out why. The container logs show no errors and at some point the stream simply…
ilmoi
  • 1,994
  • 2
  • 21
  • 45
0
votes
1 answer

how to Identify the cloud watch metrics for specific KCL in kinesis streams

we have multiple kinesis consumer applications(KCL 2.0) are consuming the data from the same kinesis stream. All the consumer is sending the metrics to cloud watch and in the cloud watch those are showing up. If i wanted to specifically understood…
Learnis
  • 526
  • 5
  • 25
0
votes
1 answer

Using KCL 2.x to get DynamoDBEvents from KinesisStream

From a KinesisClientRecord (record) that I obtained using KCL client, I’m able to get JSON object representing ddb update event from the stream by doing: String recordData = StandardCharsets.UTF_8.decode(record.data()).toString(); JSONObject…
Akshit Arora
  • 19
  • 6
0
votes
1 answer

Can not override DynamoDB endpoint for Java Kinesis Consumer

I Can not set up my local environment through java-aws-sdk, localstack and java-aws-kcl. After creating the consumer and trying to run it on my local environment I am getting an error that my credentials are incorrect. So Kinesis consumer always go…
Vissu
  • 11
  • 1
0
votes
1 answer

AWS KCL with MultiLangDaemon: Checkpointing to "LATEST"?

I have a Kinesis consumer whose job is to keep track of "currently active users" in a system. Every minute users send a heartbeat to a Kinesis stream and this system just keeps a list of all unique user GUIDs it has seen along with the last time…
stevendesu
  • 15,753
  • 22
  • 105
  • 182
0
votes
0 answers

How to read environment variables in properties file for Python client for Kinesis Client Library (KCL)?

Is there a way to read the environment variable inside the .properties file used for the configuration of KCL workers? Specifically, I want to know for streamName attribute. Or if it is not possible through the .properties file, can I configure KCL…
maro
  • 67
  • 6
0
votes
1 answer

KCL 2.x Enhanced Fan Out CloudWatch Metrics (MillisBehindLatest vs SubscribeToShardEvent.MillisBehindLatest)

was wondering if somebody could clarify the difference between these two metrics when used in an enhanced fan-out (KCL 2.x) application. I believe the KCL 'MillisBehindLatest' measure the iterator age used in the GetRecords call. However, when…
John Doe
  • 205
  • 6
  • 17
0
votes
1 answer

Does Kinesis Client Library consumer support AT_TIMESTAMP as a starting position when using Dynamodb streams kinesis adapter

According to doc Dynamodb streams and Kinesis data streams low-level APIs are similar, but they are not 100 percent identical. I notice that Dynamodb streams' GetShardIterator is a bit different i.e it doesn’t support AT_TIMESTAMP as a shard…
reda la
  • 801
  • 1
  • 6
  • 10
1 2 3 4
5
6 7