Stack Overflow
Stack Overflow

Questions tagged [amazon-kcl]

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

Source: https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html

99 questions
0
votes
0 answers

Resume reading from kinesis after a KCL consumer outage

I would like to understand what are the best ways of handling a kinesis consumer outage. My consumer is set up with TRIM_HORIZON. Below is the definition from AWS: TRIM_HORIZON - Start reading at the last (untrimmed) stream record, which is the…
moulay
  • 1
0
votes
0 answers

How do we process the kinesis stream data every one hour?

I have a kinesis stream which is being continuously written with putRecord. In the consumer I consume using processRecords of KCL. This process records gets called whenever there is a record in the stream.I need this process records to be excuted…
alexgids
  • 396
  • 3
  • 11
0
votes
0 answers

Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/services/kinesis/clientlibrary/interfaces/IRecordProcessorFactory

I am developing an Kinesis application which is putting Kinesis Stream data to S3, It is internally using KCL (Kinesis Client Library).It is working fine in Eclipse when i am running it on my local machine. I exported this project in jar file. While…
Awadesh
  • 3,530
  • 2
  • 20
  • 32
0
votes
1 answer

Dynamodb stream order of records

I am populating records in dynamodb in the following sequence: A11, A12, A13, A14, A15, A21, A22, A23, A24, A25, A31, A32, A33, A34, A35 Records with same prefix (Ai) have the same partition key but different sort keys. Assume that all the records…
ishan3243
  • 1,870
  • 4
  • 30
  • 49
0
votes
1 answer

What is the proper 'require' for KCL in a NodeJS consumer application?

I'm reading through the docs for KCL (AWS's Kinesis Client Library) and if I'm understanding it correctly, I need to install the KCL itself (Java) and then my NodeJS consumer application will be able to access it. First of all, is this correct? If…
Steverino
  • 2,099
  • 6
  • 26
  • 50
0
votes
1 answer

using Kinesis Client library with Spark Steaming PySpark

I am looking for using KCL on SparkStreaming using pySpark. Any pointers would be helpful. I tried few given by spark Kinesis Ingeration link. But i get the error for JAVA class reference. Seems Python is using JAVA class. i tried…
Giri
  • 35
  • 5
0
votes
1 answer

What are the options to process timeseries data from a Kinesis stream

I need to process data from an AWS Kinesis stream, which collects events from devices. Processing function has to be called each second with all events received during the last 10 seconds. Say, I have two devices A and B that write events into the…
xtx
  • 4,306
  • 4
  • 28
  • 30
0
votes
2 answers

How does Kinesis distribute shards among workers?

Is there any attempt to keep adjacent shards together when spreading them out over multiple workers? In the documentation example it started with 1 worker/instance and 4 shards. Then auto-scaling occurred and a 2nd worker/instance was started up.…
darrickc
  • 1,872
  • 6
  • 27
  • 38
-1
votes
1 answer

aws KCL(Kinesis Client Library) library is throwing errors when building

I am following this link to start amazon KCL, Kinesis Client Library, because I need to use nodeJ. I following steps here: https://github.com/awslabs/amazon-kinesis-client But when I run mvn clean install -Dgpg.skip=true, I got error as…
user3006967
  • 3,291
  • 10
  • 47
  • 72
1 2 3 4 5 6
7