Amazon Kinesis Data Analytics is the way to analyze streaming data, gain actionable insights, and respond in real time. SQL users can query streaming data or build entire streaming applications using templates and an interactive SQL editor. Java developers can build streaming applications using open source Java libraries and AWS integrations to transform and analyze data in real-time.
Questions tagged [amazon-kinesis-analytics]
133 questions
0
votes
1 answer
Apache Flink - Checkpointed Data Size is increasing over the period of time
I am using Event time semantics in my Flink application (version 1.11.1) which is running in AWS - kinesis analytics. This application has source as kinesis stream and sink as Postgres. Checkpointing interval is 10 seconds as DB sink is triggered on…
Swapnil Khante
- 547
- 2
- 10
0
votes
1 answer
Apache Flink EventTime processing not working
I am trying to perform stream-stream join using Flink v1.11 app on KDA. Join wrt to ProcessingTime works, but with EventTime I don’t see any output records from Flink.
Here is my code with EventTime processing which is not working,
public static…
Sairam Sankaran
- 309
- 9
- 18
0
votes
1 answer
Single source multiple sinks v/s flatmap
I'm using Kinesis Data Analytics on Flink to do stream processing.
The usecase that I'm working on is to read records from a single Kinesis stream and after some transformations write to multiple S3 buckets. One source record might end up in…
Vipul
- 195
- 1
- 4
- 15
0
votes
0 answers
Kafka metrics not produced when running Flink in AWS Kinesis Analytics
I have an Flink application which produces Kafka messages. To do so, I use the Apache Kafka connector.
This application is running using AWS Kinesis Analytics.
The application is running smoothly and the messages are successfully produced to the…
Thierry Nowak
- 206
- 2
- 6
0
votes
1 answer
Issue with AWS Kinesis SQL - Random Cut Forest algorithm
I have this code in an AWS Kinesis application:
CREATE OR REPLACE STREAM "OUT_FILE" (
"fechaTS" timestamp,
"celda" varchar(25),
"Field1" DOUBLE,
"Field2" …
Jose Rondon
- 370
- 2
- 6
- 13
0
votes
2 answers
Start a Kinesis data analytics application with cloudformation
Is there a way to start a kinesis data analytics application via cloudformation?
In these instructions, under the Start the Application section, there are instructions for how to start a Kinesis Data Analytics application through the console or…
vbp13
- 1,040
- 1
- 10
- 20
0
votes
1 answer
QueryableState Flink AWS EMR
I have a use-case of using Queryable state feature in an Apache Flink cluster.
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#activating-queryable-state
I do not want to setup my own cluster (but want…
0
votes
1 answer
Kinesis Analytics Session or Stagger Window Batching Without Aggregation
I'm looking to use Kinesis Data Analytics (or some other AWS managed service) to batch records based on a filter criteria. The idea would be that as records come in, we'd start a session window and batch any matching records for 15 min.
The stagger…
brandonhilkert
- 4,205
- 5
- 25
- 38
0
votes
1 answer
How to Parse Json in Kinesis Analytics SQL query
How to Parse Json in Kinesis Analytics SQL query.
I have streaming data received from Kinesis Stream, in column I have json wanted to ready some elements of Json
Ex. In column body I have below Json
{"deviceStatus":"Active", "deviceId":"11111"}
I…
Saurabh Raoot
- 1,303
- 3
- 26
- 31
0
votes
1 answer
Limiting Network Traffic in Flink with Kinesis
I have a Flink application running in Amazon's Kinesis Data Analytics Service (managed Flink cluster). In the app, I read in user data from a Kinesis stream, keyBy userId, and then aggregate some user information. After asking this question, I…
ChrisATX
- 109
- 1
- 11
0
votes
2 answers
Flink - External Checkpoints on AWS
I plan to use Flink on AWS Kinesis Analytics for Java Applications to perform stateful streaming aggregation.
I'd like to save checkpoints to a persistent store. What are my options?
Can I use S3 using FsStateBackend?
What about RocksSB? Is…
user544192
- 695
- 2
- 10
- 23
0
votes
1 answer
How to unit test a KinesisRecord-based DoFn?
I’m getting started on a Beam project that reads from AWS Kinesis, so I have a simple DoFn that accepts a KinesisRecord and logs the contents. I want to write a unit test to run this DoFn and prove that it works. Unit testing with a KinesisRecord…
Alec
- 490
- 1
- 4
- 8
-1
votes
1 answer
Flink-Kafka Flink job reading kafka records during startup and failing to start on AWS-KDA
Running a Flink-Beam job on KDA (kakfa --> flink(beam) --> ElasticSearch) the simple job wont start on KDA and goes to infinite loop. The AWS KDA Support replied saying the Job reads records during startup which is the cause of failure.
The…
aditya kumar
- 9
- 4