Stack Overflow
Stack Overflow

Questions tagged [amazon-kcl]

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

Source: https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html

99 questions
2
votes
2 answers

How to replay Kinesis records with KCL C#

I want to Replay Kinesis Records by a given Timespan, (or a start time) use case: on application failure, some stream records might be considered processed, but not actually being finalized. i.e. not being saved in local data store. so on system…
D.J
  • 2,534
  • 4
  • 28
  • 43
2
votes
2 answers

Error: Could not find or load main class com.amazonaws.services.kinesis.multilang.MultiLangDaemon

what's the problem?? i follow this tutorial to run the amazon kinesis. https://github.com/awslabs/amazon-kinesis-client-python i follow until the last step, that will give me the command line.this is my command line c:\Program…
john
  • 21
  • 4
1
vote
0 answers

How to slow down reads in Kinesis Consumer Library?

We have an aggregation system where the Aggregator is an KDA application running Flink which Aggregates the data over 6hrs time window and puts all the data into AWS Kinesis data Stream. We also have an Consumer Application that uses KCL 2.x library…
cool_fire
  • 93
  • 1
  • 6
1
vote
1 answer

Timeout issues when using Amazon Kinesis Client Library resulting in dropped records

I am facing the following issue when running a KCL consumer against a LocalStack instance: [INFO ] 2021-07-15 17:30:34.019 [software.amazon.kinesis.coordinator.DiagnosticEventLogger][task-1] DiagnosticEventLogger - Current thread pool executor…
Eugene Shvartsman
  • 185
  • 10
1
vote
0 answers

Amazon kinesis KCL 1.14.3 - multiple shards and instances issue

I have a two instances of my app and two shards in kinesis. Scenario, Launching instance 1. Application gets two shards. Everything is ok. Launching instance 2 after a few time laters. When Instance 1 processing records from both shards then in…
Mateusz
  • 353
  • 2
  • 15
1
vote
1 answer

Could not initialize class KinesisClientLibConfiguration while integrating kinesis consumer

I am integrating kinesis in my java springboot project, i am able to publish the data into kinesis stream but while consuming it i am getting error like below : Could not initialize class…
Mohammad Wasim Khan
  • 95
  • 1
  • 1
  • 9
1
vote
1 answer

Different credentials for Kinesis Stream, DynamoDB and CloudWatch inside Spring Cloud Stream

I am using Spring Cloud Stream Kinesis binder (version 2.1.0) Because of security reasons, I must have one set of credentials for Kinesis and another set of credentials for DynamoDB and CloudWatch. Everything works fine if…
Marin Krešo
  • 49
  • 3
1
vote
1 answer

How does LATEST position in stream works in Kinesis, KCL?

We are building a service based on Kinesis / DynamoDB streams and we have the following question about the behavior of the checkpoints. We have a worker that starts with the following configuration withInitialPositionInStream…
Joaquín Fernández
  • 147
  • 1
  • 4
  • 19
1
vote
1 answer

Can I use one lease table for multiple KCL applications?

We are building a service based on Kinesis / DynamoDB streams and one question we have (which we cannot find in the official documentation) is whether we can use the same lease table (DynamoDB) to store checkpoint information of different KCL…
Joaquín Fernández
  • 147
  • 1
  • 4
  • 19
1
vote
1 answer

Consume records from specific shards under KCL 2.x ( Kinesis )

I have set of records under some specific shards in the Kinesis stream. I m using KCL 2.x consumer to consume records from kinesis, but the issue is that consumer is fetching me records from all the shards available in the stream. So is there any…
Hidden Coder
  • 55
  • 3
1
vote
1 answer

How Apache Beam manage kinesis checkpointing?

I have a streaming pipeline developed in Apache Beam (using Spark Runner) which reads from kinesis stream. I am looking out for options in Apache Beam to manage kinesis checkpointing (i.e. stores periodically the current position of kinesis stream)…
Viswa
  • 1,357
  • 3
  • 18
  • 30
1
vote
0 answers

KCL lease table is not created in dynamo-db after consumer is connected

We have newly created kinesis stream, which causes the below error while trying to connect via KCL in java and the stream is available to put records on the fly. java.lang.RuntimeException: …
Mukundhan
  • 3,284
  • 23
  • 36
1
vote
1 answer

KCL implementation with Docker containers

I have a use case in which there is an existing Kinesis stream of which i am not aware of how many shards are there . However i have to create a consuming application which would consume from the shard and i also have to Dockerize the application. I…
Subhayan Bhattacharya
  • 5,407
  • 7
  • 42
  • 60
1
vote
2 answers

What IAM permissions does a Kinesis Consumer need when using KCL?

I have a Kinesis consumer I wrote using the Kinesis Client Library (KCL). This consumer is running under an assumed IAM role. I've read from the documentation that: The KCL creates a DynamoDB table with the application name and uses the table to…
Kenny Worden
  • 4,335
  • 11
  • 35
  • 62
1
vote
1 answer

Is there a way to consume sample records from kinesis?

I wrote an Amazon Kinesis consumer using KCL, which manages my record processing task. It is currently processing the records without filtering. I am looking for a way to process sample records while skipping some of them. For example, in case…
Nadav Miran
  • 59
  • 1
  • 5
1 2
3
4 5 6 7