Stack Overflow
Stack Overflow

Questions tagged [amazon-kcl]

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

Source: https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html

99 questions
1
vote
0 answers

AWS KCL and handling errors

The case. KCL reads a batch of 500 records from kinesis. Something bad happens (e.g. Unexpected error). KCL will lose all 500 records with data. Because KCL marks these 500 records as read and handled even though errors occur. If I use AWS Lambdas…
Bogdan Surai
  • 1,197
  • 14
  • 16
1
vote
1 answer

Change failover time for aws kcl

AWS recommends to increase failover time for KCL (kinesis), if apps with connectivity issues. https://docs.aws.amazon.com/streams/latest/dev/troubleshooting-consumers.html But I can’t find how failover time can be changed. I’m looking for (one or…
Bogdan Surai
  • 1,197
  • 14
  • 16
1
vote
0 answers

EMR vs KCL on aws Lambda

We are planning to consume messages from Kinesis stream. We need to process event data one by one (no windowing, no aggregation), and we could see multiple options to read data from kinesis stream: Kinesis consumer library Lambda function…
Pragmatic
  • 3,093
  • 4
  • 33
  • 62
1
vote
1 answer

KCL application does not take all shards while consuming multiple streams

I have a consumer application developed using KCL. application creates a Worker for each stream and each stream has more than one shard. In different executions, application takes different number of shards and it is not taking all shards. I am…
sansari
  • 558
  • 1
  • 7
  • 17
1
vote
0 answers

AWS KCL C# not processing records

So I am new to AWS Kinesis, and I have a basis producer working just fine with using my default profile which has my IAM user details in the store. I can see that the producer send messages to the single stream shard. That's all good. I know there…
sacha barber
  • 2,214
  • 1
  • 24
  • 37
1
vote
1 answer

AWS KCL capture shutdown signal in docker

When I ran the KCL as ./bin/kcl-bootstrap -p app.properties -j /usr/bin/java -e at the terminal, and then I did ps -ef | grep java and got the pid for it. I issued kill -s TERM $PID_ABOVE. I noticed the node recordProcessor managed to log like this…
Tin Ng
  • 937
  • 2
  • 11
  • 25
1
vote
1 answer

Kinesis Client Library for Consumers - Managing Dynamo DB tables

We have multiple consumers reading from the same stream. So everytime there is new consumer, a new table is created in DynamoDB. These consumers are not permanent. Once they are shutdown, the table becomes redundant. 1.If we keep the application…
FunWithJava
  • 213
  • 1
  • 2
  • 9
1
vote
1 answer

Maintaining order using Amazon's Kinesis with atleast 1000 messages per sec

I am trying to send 1000's of messages in sequence using Amazon's Kinesis but messages being received by kinesis consumer is not in sequence. Things I have tried : Used putRecord() method to send with SequenceNumberForOrdering. Achieved ordered…
superB
  • 288
  • 2
  • 13
1
vote
1 answer

When AWS KCL processRecords is failed, how to "mark" that the records should be reprocessed?

I'm working with AWS DynamoStream which his API is based on the AWS KCL. In cases I received records which I failed to process and I want those records to be available later to allow reprocessing of them. For instance I'm trying to save them to a…
Roee Gavirel
  • 18,955
  • 12
  • 67
  • 94
1
vote
0 answers

How to alter kinesis consumer properties in spark streaming

Here is the KinesisReceiver documentation. Here all the default values of KinesisClientLibConfiguration are used. Is there a way to change the default values using a properties file in spark streaming application?
Sarath Chandra Jiguru
  • 135
  • 1
  • 1
  • 10
1
vote
1 answer

Will Kinesis Client Library DynamoDB Adaptor Lose Data

The DynamoDB Streams Kinesis Adaptor published on github here has this function with the following comments: The Kinesis model provides an adjacent parent shard ID in the event of a parent shard merge. Since DynamoDB Streams does not…
Lim
  • 329
  • 2
  • 15
1
vote
1 answer

exception when python KCL connects to kinesis stream

I am trying to integrate kinesis in spark streaming and for that I am using python and KCL. I get this exception most of the times when reading from kinesis 'utf8' codec can't decode byte 0xf1 in position 940: invalid continuation byte Can someone…
Nipun
  • 4,119
  • 5
  • 47
  • 83
1
vote
1 answer

Can Amazon's Kinesis Client Library consume multiple streams?

I have a quick question. Is the KCL able to consume from multiple streams? Should you ever set up multiple streams for your application, or is a individual stream supposed to be tied with an individual application? My particular use case is that I…
ScottBouloutian
  • 59
  • 2
  • 6
0
votes
0 answers

Working With KCL 2.x is causing issue while "Syncing Kinesis shard info"

I am using AWS SDK V2 with KCL 2.x. I have enabled dynamodb table with streams. While creating the KCL Scheduler I am using the below configuration TableDescription tableDescription = …
user3186696
  • 1
  • 1
0
votes
1 answer

How to shutdown processRecord method of KCL in case of any error?

I am using multiple Scheduler to process record from multiple kinesis stream. Now, my requirement is, If scheduler I failed during processing, so I want to stop it, and don't want to stop scheduler II. But currently, processRecord method is not…
Prathviraj Singh Chouhan
  • 43
  • 1
  • 8
1 2 3
4
5 6 7