Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-kpl]

52 questions
2
votes
0 answers

How to control number of records in Kinesis Client library

I am trying to control number of records fetched in processRecords method of KCL, here is my KCL configuration new KinesisClientLibConfiguration( config.getApplicationName(), config.getStreamName(), awsCredentialsProvider, workerId) …
Amit Kumar
  • 825
  • 1
  • 9
  • 19
2
votes
1 answer

way to set the timing of sending the data in kinesis analytics for example every 5 mins

I am really confused about how kinesis and kinesis analytics work. I need to send some data to kenisis analytic and then the top ten stories(data) should be sent to a lambda function every 5 minutes. I have Kinesis stream and also kinesis analytics…
Hamed Minaee
  • 2,480
  • 4
  • 35
  • 63
2
votes
0 answers

Getting error while running amazon-kinesis-producer-sample

I tried running amazon-kinesis-producer-sample I used amazon-kinesis-producer version 0.10.2 on Windows7. Getting below error: 2017-03-30 14:44:41 [pool-2-thread-2] ERROR c.a.s.k.producer.KinesisProducer - Error in child…
swatik
  • 21
  • 4
2
votes
2 answers

Kinesis lambda DynamoDB

I am learning the AWS services for a use case. After going through the docs I came came up with the a simple flow. I want to ingest data into the Kinesis streams by using the Streams API and the KPL. I use the example putRecord method to ingest data…
Dan
  • 801
  • 2
  • 14
  • 29
1
vote
1 answer

Could not initialize class KinesisClientLibConfiguration while integrating kinesis consumer

I am integrating kinesis in my java springboot project, i am able to publish the data into kinesis stream but while consuming it i am getting error like below : Could not initialize class…
Mohammad Wasim Khan
  • 95
  • 1
  • 1
  • 9
1
vote
0 answers

No need to Dynamodb client while consuming data from Kinesis using kcl

I am using kcl api version 2 ,and dont want to use Dynamodb Client for storing the records . private static final Logger LOG = LoggerFactory.getLogger(DisplayConsumerApplication.class); public static void main(String... args) { …
Chintamani
  • 1,076
  • 7
  • 23
  • 47
1
vote
1 answer

Kinesis Producer callback functions - guaranteed delivery?

Streaming to Kinesis billions of messages a day. We're looking for an implementation that would allow us to deliver messages to Kinesis with exactly-once guarantee. Our producer framework requires a streaming sink to be idempotent for exactly-once…
Tagar
  • 13,911
  • 6
  • 95
  • 110
1
vote
1 answer

How secure is Aamazon Kinesis endpoint? Does the Kinesis Producer Library (KPL) securely sends messages to Kinesis endpoint?

I am building Amazon Kinesis based streaming data ingestion from on-prem data source. I can use the Kinesis Producer Library (KPL) to produce messages to Kinesis Data Stream. I wanted to know how secure is the data transfer/ingestion through KPL. Is…
Atul Dambalkar
  • 111
  • 1
  • 5
1
vote
0 answers

How to convert/decode data.Payload from kinesisvideomedia.getMedia request in AWS to an mkv file?

I'm trying to getMedia from an AWS kinesis instance which is working (as in I can stream video to it). Once I have saved the data which comes in from the getMedia call, in my nodejs code. I'm trying to convert the data.Payload which is of type…
bob123
  • 11
  • 2
1
vote
1 answer

Kinesis creates multiple records with the same sequence number

Based on Kinesis documentation, sequence number is supposed to be unique, however we see the same value being reused across multiple records. Our event producer is Spring Boot application that uses KPL internally, consumers are AWS lambdas. We have…
Andriy Kharchuk
  • 1,165
  • 1
  • 13
  • 25
1
vote
0 answers

Flink kinesis connector crashes with daemon exception

I am using flink 1.7.2 on debian slim, and kubernetes as my resource manager. But when i deploy it, it works for an hour or so without any issues and then starts failing with the following error: java.lang.RuntimeException: Exception occurred while…
indraneel
  • 405
  • 4
  • 10
1
vote
0 answers

Does AWS Kinesis Producer Library aggregate data in memory?

AWS Kinesis Producer Library may be configured to aggregate records before sending to AWS Kinesis Stream. For example we may set: final KinesisProducerConfiguration config = new KinesisProducerConfiguration(); …
mpr
  • 3,730
  • 3
  • 17
  • 22
1
vote
1 answer

Getting "Shard map update for stream xxx" failed using Amazon KPL: (AWSErrorMarshaller)Encountered Unknown AWSError

I'm trying to write to an Amazon Kinesis Firehose using the KPL library. Despite my best efforts, every attempt to connect and write to the library results in the following log lines: 2018-09-11 15:23:12.369 INFO 79765 --- [kpl-daemon-0003]…
wiltonio
  • 121
  • 2
  • 5
1
vote
1 answer

AWS Kinesis KPL behaviour when incoming data exceeds stream capacity

Suppose i have a single shard Kinesis stream which allows 1MBps incoming. I have a producer using KPL which is trying to pump 2MBps (using async call of KPL). Given the KPL behaviour of retrying, i would assume it would buffer events for some time…
dy10
  • 117
  • 1
  • 8
1
vote
1 answer

Amazon Kinesis - Identify Step Timeout

Trying to create a Kinesis Analytics query to alert when a step in a process has taken too long (or died and not moved on). I have a steam of data that contains status updates as a multi-step process moves from step to step. I am trying to write a…
Tim Merkel
  • 13
  • 4
1
2
3 4