Amazon Kinesis Data Analytics is the way to analyze streaming data, gain actionable insights, and respond in real time. SQL users can query streaming data or build entire streaming applications using templates and an interactive SQL editor. Java developers can build streaming applications using open source Java libraries and AWS integrations to transform and analyze data in real-time.
Questions tagged [amazon-kinesis-analytics]
133 questions
1
vote
1 answer
Creating Kinesis Analytics applications using aws cli
I want to create a kinesis analytics application using aws cli. I use this command to create the application
aws kinesisanalytics create-application --application-name smartfactorytest1 --application-code "CREATE OR REPLACE STREAM…
Anshul Walia
- 547
- 1
- 5
- 9
1
vote
0 answers
Kinesis Analytics Tumbling Window to Group Common Events Together
I have a use case where Kinesis is ingesting events from video streams (each video stream has a unique ID), however the data broken up and is arriving out of order.
For example:
VideoId1 {}
VideoId2 {}
VideoId2 {}
VideoId1 {}
VideoId1 {}
What I'm…
Mmm Donuts
- 9,551
- 6
- 27
- 49
1
vote
1 answer
What is a monotonic column?
I've read the definition here https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sql-reference-monotonic-expressions-operators.html, however it's still unclear to me. I understand that monotonic means ascending or descending. Does that mean…
Mmm Donuts
- 9,551
- 6
- 27
- 49
1
vote
0 answers
how to read data from kinesis analytics using Lambda function in node js?
I'm trying to read data from kinesis analytics using Lambda function in node JS,
I'm getting data in an encoded format when I'm trying to decode it I'm not getting any output.
console.log('Loading function');
const zlib =…
saran k
- 307
- 2
- 14
0
votes
0 answers
AWS Kinesis streaming application failed to sink data to Redis
I am using AWS Kinesis Streaming Application to process streaming data.
The application is running but I found nothing was sank to my Redis. Related code:
public class SpyProcessor {
private static final Logger log =…
0
votes
0 answers
Refreshing Kafka security certificates in Apache Flink application running on Amazon KDA
I am building a Flink Application and currently planning on deploying using Amazon Kinesis Analytics. I need to connect to a Kafka topic using SSL truststore/keystore certificates and am looking at using Flink's out of the box Kafka connector. It…
Daniel Allen
- 3
- 2
0
votes
0 answers
Most suitable architecture for AWS realt-time ETL pipeline with transactional tables as a sink
The task was as following - build a near real-time pipeline for the project in AWS infrastructure. Having Nosql(DynamoDB) and SQL(RDS databases) (S3 stored data can be added in the future) sources we need to combine them into tables, used by…
ground_control
- 26
- 3
0
votes
0 answers
How to upgrade Amazon Kinesis Analytics Apache Flink runtime?
We have an application running on Apache Flink 1.13, presume we have a branch we have updated the code to Flink 1.15.2 and built a jar and put that on S3.
What do we do in the AWS Console to change an existing KDA "Streaming application"'s Flink…
samthebest
- 30,803
- 25
- 102
- 142
0
votes
1 answer
How to add multiple consumer of same data stream in Flink
I am struggling with implementing solution where the problem is as below.
There is one Kinesis stream named sushistream, event comes from the stream would contains two things, i.e. broadcastId and eventID. Now I want sliding window count for every…
Dipesh Darji
- 3
- 1
0
votes
0 answers
High CPU Consumption using Cassandra Sink in Kinesis Analytics
I'm trying to set up a Kinesis Analytics application to write data from a Kinesis Stream to Keyspaces. To do this, I'm using flink's cassandra connector.
The application receives messages from a stream, groups them on some key, aggregates them over…
maxwellray
- 99
- 7
0
votes
0 answers
"Early return" for WindowedStream aggregate or reduce
tl;dr is there a way to emit the first element of a WindowedStream which matches some condition, and to stop processing all events in that window for a given key after the condition has been matched?
I am performing a reduce operation on a…
Jake
- 321
- 3
- 12
0
votes
0 answers
Apache Flink on python UDFs are not working. Despite following AWS repo on pyflink UDFs
I see AWS has a Kinesis-DataAnalytics stream with Apache Flink. And on the pyflink library for python there is a way to create UDF. In fact, AWS seems to have a repo with examples for pyflink UDF.
I have tried following the guide, and doing the UDF…
Eugenio.Gastelum96
- 164
- 1
- 13
0
votes
0 answers
Kinesis Analytics SQL application does not deliver all the stream output data to destination lambda
I have a Kinesis Analytics SQL application (legacy) that computes most frequent items (top 10) in 1 minute window using TOP_K_ITEMS_TUMBLING function:
CREATE OR REPLACE STREAM "TOP_N_STREAM"
("myItem" VARCHAR(256), "frequency" BIGINT);
CREATE OR…
revy
- 3,945
- 7
- 40
- 85
0
votes
0 answers
Records not showing in Flink when watermarking is activated
Trying to run the a flink application locally with a local Kinesis stream.
The following code works perfectly (as in, records can be seen in the sink table path), but when I change the watermark from event_ts to event_ts - INTERVAL '10' SECOND, the…
SinkingTitanic
- 181
- 1
- 12
0
votes
0 answers
Properly deploying an AWS Kinesis Application
I'm trying to setup a cicd process for my kinesis applications which have been developed using apache beam running on flink.
My kinesis application is setup to restore the latest snapshot and it stores an internal state.
However, I noticed that…
Stefn
- 94
- 6