Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-kpl]

52 questions
1
vote
0 answers

Unable to write data to Amazon Kinesis from Linux server

I have deployed my java job on a linux server to consume data from various APIs and publish it to Kinesis. My job is unable to open up the Kinesis stream to publish the data and throws below error. Any suggestion would be of great help. 31253…
devop.sashi
  • 13
  • 3
0
votes
0 answers

How can I send record as a batch to kinesis stream using kpl when application shuts down/goes to sleep?

About the application: I have a spring application deployed to AWS Fargate Service. This application consumes data from SQS queue. Then, apply some logic and send those data to Kinesis Stream. The application gets extreme amount of request at some…
Rue Honda
  • 13
  • 3
0
votes
0 answers

logically Isolating data in apache Flink

We are processing around 30 million of records per day using a apache flink job, this flink job filters the data data from source kinesis stream and push the filtered data to the respective kinesis streams which are on some other AWS account, here…
Yogesh Katkar
  • 133
  • 10
0
votes
0 answers

multiple consumers reading from same shard in AWS Kinesis

When multiple consumers are reading from same shard in AWS Kinesis, when a consumer reads a message, is that message deleted and not accessible for other consumers? I have multiple consumers for a kinesis shard, when the second consumer is making…
abhinav
  • 527
  • 3
  • 11
  • 24
0
votes
0 answers

Not utilising the full capacity of Amazon Kinesis Data Stream

I am sending data to kinesis data stream using putRecords request asynchronously (500 records every time). There are 4 shards in the stream in which multiple putRecords requests continuously sends data using a for loop. But when I checked the stream…
Govind Yadav
  • 33
  • 4
0
votes
1 answer

Streaming data from Kinesis to S3 fails with Illegal Character that KPL itself writes

I have a relatively straightforward use case: Read Avro data from a Kafka topic Use KPL (v0.14.12) to send this data to Kinesis Data Streams Use Kinesis Firehose to transform this data into Parquet and transfer it to S3. The Kafka topic was…
filpa
  • 3,651
  • 8
  • 52
  • 91
0
votes
1 answer

Unable to read kinesis data stream from Node App

I have setup a kinesis data stream to capture RDS changes and adding data to this stream. Data is feeding to this stream as i can see from stream dashboard. But I'm unable to create consumer, I tried KCL sample code for node but it just keep running…
Basit
  • 862
  • 1
  • 12
  • 30
0
votes
1 answer

Spring Cloud Stream Kinesis binder error in child process

I would like to use Spring Cloud Stream kinesis binder with KPL/KCL enabled. However, when I enabled that by using kpl-kcl-enabled: true the following error keeps coming up: com.amazonaws.services.kinesis.producer.IrrecoverableError: Error starting…
Ali
  • 1,759
  • 2
  • 32
  • 69
0
votes
1 answer

How to add tags to aws resources created by spring stream cloud with kinesis binder and kcl enabled

I have spring cloud stream application with kinesis binder The application can create resources on startup if not exists: kinesis streams, dynamodb tables for kcl I would like that on creation of those resources i will be able to add custom tags to…
Panteleev Dima
  • 175
  • 2
  • 18
0
votes
1 answer

AWS Kinesis KCL skips records added before startup

I started to use both KPL and KCL to exchange data between services. But whenever consumer service is offline, all data sent by KPL are lost forever. So I get only those chunks of data that were sent while consumer service is up and its…
Alesto
  • 639
  • 4
  • 13
  • 29
0
votes
1 answer

spring cloud stream kinesis configuration

I am trying integrate spring cloud stream kinesis in my app but i cant find all configuration option in there manual. I have seen this…
Mahender Singh
  • 1,393
  • 3
  • 17
  • 23
0
votes
1 answer

Using Kinesis producer library in Spark

I have Spark job reads millions of records from Cassandra, filter out(business rules) and write to Kinesis stream. I don't find any example and testimonial on how to invoke KPL(Kinesis Producer Library) from Spark. Is that correct approach? Do I…
ameenun
  • 1
  • 1
0
votes
1 answer

Failed to put record in kinesis by lambda

Since I am new to lambda and kinesis so apologies in advance. I have an SQS subscribed to SNS. A lambda polls the message from SQS and publish it to kinesis after XYZ processing. I want to know if putting record to kinesis by lambda fails what is…
SAA
  • 45
  • 7
0
votes
0 answers

Connecting kinesis analytics to lambda direcltly and also indirectly by using kinesis stream results in different output

I am facing a very strange problem and I am not sure if it is a bug in aws or my lack of understanding. Here is the problem. I have a kinesis analytins reoprting every 40 seconds. so lets say it reported the following 40 second ago: {frist row with…
Hamed Minaee
  • 2,480
  • 4
  • 35
  • 63
0
votes
1 answer

KPL AWS Kinesis Producer library failure without error messages

I'm running KPL inside dropwizard. Following is my code. AWSStaticCredentialsProvider credentialsProvider = new AWSStaticCredentialsProvider(new BasicAWSCredentials("XXXX", "XXXXXXXXXXXX")); kinesis = new…
ravi404
  • 7,119
  • 4
  • 31
  • 40
1 2
3
4