Amazon Kinesis Data Analytics is the way to analyze streaming data, gain actionable insights, and respond in real time. SQL users can query streaming data or build entire streaming applications using templates and an interactive SQL editor. Java developers can build streaming applications using open source Java libraries and AWS integrations to transform and analyze data in real-time.
Questions tagged [amazon-kinesis-analytics]
133 questions
1
vote
0 answers
Combining dynamodb streams
I need to build a service using aws tools which aggregates data from various dynamodb tables and stores the data in a redshift cluster. There also needs to be processing done to each data stream before it is stored in redshift.
My current idea is to…
szatz
- 11
- 3
1
vote
2 answers
Windowing is not triggered when we deployed the Flink application into Kinesis Data Analytics
We have an Apache Flink POC application which works fine locally but after we deploy into Kinesis Data Analytics (KDA) it does not emit records into the sink.
Used technologies
Local
Source: Kafka 2.7
1 broker
1 topic with partition of 1 and…
Peter Csala
- 17,736
- 16
- 35
- 75
1
vote
0 answers
AWS Kinesis Data with Base64
I have a kinesis stream which has data with its original value as Base64 encoded . As Kinesis by default encodes all its data to Base64, I have to do a Base64decode
to getdata from the Kinesis DataBlob . But since the originalData is also Base64…
Nephy
- 11
- 4
1
vote
1 answer
AWS Flink (KDA) rocks db checkpoint size is increasing and never comes down
In our application , flink checkpoint size is increasing and never comesdown with rocks db as statebackend.(AWS KDA)
Kafka-->do some magic -->ES(Sink)-->Writes to kafka
here the keys we use are UUID and are never repeated. How can i configure to…
Fryder
- 413
- 2
- 7
- 21
1
vote
1 answer
How to perform multiple window aggregation with update in Flink?
I have a use case wherein I'm receiving a stream of events containing different sets of information and want to perform aggregations on them. For each of these aggregations, there are multiple tumbling windows which are needed eg: Daily, Weekly,…
shashankg77
- 2,206
- 4
- 18
- 34
1
vote
1 answer
Consuming DynamoDB Streams with AWS Kinesis Data Analytics
I am using Kinesis Firehose to consume Dyanamo DB streams through lambda and pushing those records to S3 bucket, Glue job is running every hour to pick the records from S3 , perform deduplication and then finally insert the records to Redshift.
Is…
Vicky
- 1,135
- 1
- 17
- 37
1
vote
1 answer
Website Click stream flow + customer 360 using AWS Kinesis Firehose
We are trying to implement a clickstream flow of our e-commerce on AWS. The clickstream will catch all actions done by 'Anonymous' users. Anonymous users are tracked via a UUID , generated during their first visit , that is stored in a cookie. We…
Lina
- 1,217
- 1
- 15
- 28
1
vote
0 answers
How to troubleshoot Kinesis Analytics SQL application?
I have a KA SQL application deployed via CloudFormation. It is meant to read the data of the kinesis stream then run simple SQL count query and output data to Firehose. Firehose in its turn should write this data to S3. Unfortunately, the data is…
Alua K
- 398
- 1
- 3
- 18
1
vote
0 answers
How to do a PERCENT_RANK in KINESIS ANALYTICS
I am trying to learn how to do simple analysis such as calculating the percentile rank of price volatility in real-time with Kinesis Analytics ,
I calculate the price volatility of each Ticker over 1 minute sliding window and then try to percent…
sgnab
- 41
- 2
1
vote
0 answers
No need to Dynamodb client while consuming data from Kinesis using kcl
I am using kcl api version 2 ,and dont want to use Dynamodb Client for storing the records .
private static final Logger LOG = LoggerFactory.getLogger(DisplayConsumerApplication.class);
public static void main(String... args) {
…
Chintamani
- 1,076
- 7
- 23
- 47
1
vote
1 answer
aws kinesis data analytics application (flink) change property originally located at flink-conf.yaml
As the runtime of my flink application I use managed flink by AWS (Kinesis Data Analytics Application)
I added functionality (sink) for write processed events from kinesis queue in S3 in a parquet format.
Locally everything works for me, but when I…
Sergey Postument
- 167
- 12
1
vote
0 answers
Kinesis data analytics sql application is not writing logs into cloudwatch
I created a kinesis data analytics application(using SQL) and attached cloudwatch logging option.
when i run the application, i am receiving the result based on my requirements.
Problem: my kinesis-data-analytics application is not writing logs into…
siva
- 549
- 7
- 25
1
vote
1 answer
Cannot aggregate an infinite stream when using custom timestamp for thumbling window in Kinesis Data Analytics
In the documentation provided by AWS they provide examples for doing tumbling and sliding windows based on the ROWTIME which is the timestamp when Amazon Kinesis Analytics inserted the row in the first in-application stream.
The provided…
xtra
- 1,957
- 4
- 22
- 40
1
vote
0 answers
Kinesis data analytics cannot use same stream twice for different destinations
I'm using kinesis data analytics to do real time analytics.
After a couple of operations in SQL I pump the results to a sql stream named "OUTPUT_STREAM".
Next I configured a destination that routes the "OUTPUT_STREAM" to a kinesis data…
xtra
- 1,957
- 4
- 22
- 40
1
vote
2 answers
allowedLateness on Global Window custom trigger
I have created a custom trigger and processing function for my event stream.
DataStream dynamoDBRows =
sensorEvents
.keyBy("id")
.window(GlobalWindows.create())
.trigger(new MyCustomTrigger())
…
Arjun Sunil Kumar
- 1,781
- 3
- 28
- 46