Stack Overflow
Stack Overflow

Questions tagged [amazon-kcl]

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

Source: https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html

99 questions
3
votes
1 answer

How do I delay processing of AWS Kinesis messages?

I want to delay processing of messages from an AWS Kinesis stream for one hour. I've configured the KCL consumer to read a batch of records every four minutes, check the timestamp of each record, and stop processing the batch if any of the records…
landonvg
  • 41
  • 1
  • 7
3
votes
2 answers

Same Kinesis Consumer running on multiple EC2 instances

I have multiple instances of EC2 running for a same microservice, which has a Kinesis consumer running(with KCL). My question is, when Kinesis stream gets a new event, since all consumers are polling, will the same event be consumed by consumers of…
chethan
  • 35
  • 3
3
votes
2 answers

Disable INFO logs in aws kcl - Kinesis

I'm running on Ubuntu 16.04 & Java 8. kcl generate thousands of INFO log lines. Does any one know how to enable only ERROR and WARN logs? *I have also the same question for kpl. I don't have a logfile.
Stav Alfi
  • 13,139
  • 23
  • 99
  • 171
3
votes
1 answer

How do I use the requestShutdown and shutdown to do graceful shutdown in the case of KCL Java library for AWS Kinesis

I am trying to use the new feature of KCL library in Java for AWS Kinesis to do a graceful shutdown by registering with shutdown hook to stop all the record processors and then the worker gracefully. The new library provides a new interface which…
daviddecoding
  • 142
  • 1
  • 12
3
votes
1 answer

What happens if the number of workers is > number of shards when using KCL with AWS Kinesis streams?

The AWS Kinesis stream documentation mentions Typically, when you use the KCL, you should ensure that the number of instances does not exceed the number of shards What would be the consequence if the number of instances exceeds the number of…
H1101
  • 183
  • 2
  • 8
2
votes
1 answer

Multiple KCL application with same application name reading from one Kinesis Stream

I'm confused on how KCL works. First of all these are my understanding now. 1 KCL application uses one application name, creates one dynamodb table. 1 KCL application has one worker with x number of record-processor working parallel on x number of…
lai yoke hman
  • 261
  • 4
  • 12
2
votes
0 answers

Using a local instance of DynamoDB with Amazon Kinesis KCL

Is there any way to store the checkpoints / lease table on locally hosted DynamoDB? I don't see any way in KCL config to specify the connection, AWS docs also don't help. I would like to avoid forking the KCL code. If that matters, I'm using KCLPy…
Elo Siema
  • 31
  • 4
2
votes
1 answer

Multiple consumers to the Kinesis Stream using Kinesis Client Library ( KCL 2.x )

I have an application which uses KCL 2.x to consume records from Kinesis, the data present in different shards of stream is in different format and i want to process them seperately using different KCL consumers with different configurations. For…
Hidden Coder
  • 55
  • 3
2
votes
0 answers

How to control number of records in Kinesis Client library

I am trying to control number of records fetched in processRecords method of KCL, here is my KCL configuration new KinesisClientLibConfiguration( config.getApplicationName(), config.getStreamName(), awsCredentialsProvider, workerId) …
Amit Kumar
  • 825
  • 1
  • 9
  • 19
2
votes
0 answers

Spark Streaming using Kinesis doesn't work if shard checkpoints exist in DynamoDB

This was cross-posted in SPARK-22685. TL;DR – If shard checkpoints don't exist in DynamoDB (== completely fresh), Spark Streaming application reading from Kinesis works flawlessly. However, if the checkpoints exist (e.g. due to app restart), it…
Grega Kešpret
  • 11,827
  • 6
  • 39
  • 44
2
votes
3 answers

How to balance kinesis shards across several record processor?

I am currently writing the simple Kinesis Client Library (KCL) in Golang version. One of the features that I want it for my simple KCL is load balancing shards across multiple record processors and EC2 instances. For example, I have two record…
Vincent acent
  • 465
  • 2
  • 7
  • 15
2
votes
0 answers

How to set checkpoint in dynamodb streams KCL adapter for lambda that triggers every N minutes

I wanted to access DynamoDB Streams for the last N minutes using DynamoDB Streams Kinesis Adapter by triggering lambda every N minutes by scheduling CloudWatch Events. How can I keep track of last processed record when lambda is triggered next time…
Abhinav Kaushal Keshari
  • 536
  • 4
  • 9
2
votes
1 answer

Is Kinesis the right tool for my needs? (& other assorted questions)

I need to process, at peak, 100s of records per second. Those records are simple JSON bodies and they should be collected and then processed/transformed into a database. A few questions ... 1) Is Kinesis right for this? Or is SQS better suited? 2)…
mr-sk
  • 13,174
  • 11
  • 66
  • 101
2
votes
1 answer

kinesis client worker logic

I want to understand when does IRecordProcessor's processRecords method is invoked from worker. If my earlier call to processRecords is not yet completed will worker call next processRecords ? Will worker start fetching new records from kinesis or…
user1846749
  • 2,165
  • 3
  • 23
  • 36
2
votes
1 answer

How to deal with that when the processRecords is failed in AWS KCL?

I am creating an application with nodejs based on KCL. in the processRecords function, I tried to send the records which I get from kinesis to the web service with http request. But when the web service is not available, the http request will be…
Angle Tom
  • 1,060
  • 1
  • 11
  • 29
1
2
3 4 5 6 7