Amazon Kinesis Data Analytics is the way to analyze streaming data, gain actionable insights, and respond in real time. SQL users can query streaming data or build entire streaming applications using templates and an interactive SQL editor. Java developers can build streaming applications using open source Java libraries and AWS integrations to transform and analyze data in real-time.
Questions tagged [amazon-kinesis-analytics]
133 questions
0
votes
2 answers
Kinesis Application - Flink 1.11 timeout exception
I'm working with Kinesis Application using Flink 1.11, but I got the following error when start my application:
java.util.concurrent.TimeoutException: The heartbeat of TaskManager with id 421563c271e57acb4592f9d447d45b42 timed out.
at…
Felipe Jorquera Uribe
- 28
- 1
- 6
0
votes
1 answer
Temp files s3 slow down with beam (throtteling)
I'm trying to use apache beam with Flink at AWS KDA. This pipeline reads some data from kinesis and after a simple transformation, it tries to group the results into one shard using this strategy:
.apply("Creating micro-batches",…
B Berdugo
- 25
- 5
0
votes
1 answer
Can I run Amazon Kinesis Analytics (KDA) using Amazon EventBridge rule?
I need to run a KDA application using an EventBridge rule? is it possible
Yohan Kulasinghe
- 13
- 4
0
votes
1 answer
How can we Increase the Task Managers in Apache Flink application
In AWS kinesis data analytics application , integrated the Apache flink application. In apache flink dashboard Task Managers count is 2 but need to add 2 more Task Managers in apache flink for improving performance.
How can we add Task Managers in…
0
votes
0 answers
Slow flink processing
I am using Kinesis data stream as a source and elasticsearch as a sink.
I am using Flink job to process this data a little bit then sink this data to elasticsearch.
In the production environment, the Kinesis data stream can generate 50,000 events…
Rohit
- 97
- 10
0
votes
1 answer
Apache Flink with Kinesis Analytics : java.lang.IllegalArgumentException: The fraction of memory to allocate should not be 0
Background :
I have been trying to setup BATCH + STREAMING in the same flink application which is deployed on kinesis analytics runtime. The STREAMING part works fine, but I'm having trouble adding support for BATCH.
Flink : Handling Keyed Streams…
jt97
- 13
- 3
0
votes
1 answer
Apache Beam KDA application not creating checkpoints
I have an Apache Beam application deployed on Amazon KDA.
It has checkpointing enabled with the default settings.
"FlinkApplicationConfigurationDescription": {
"CheckpointConfigurationDescription": {
"ConfigurationType":…
Gayan Weerakutti
- 11,904
- 2
- 71
- 68
0
votes
1 answer
Apache flink with AWS kinesis consume data-Getting java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer
Here is my code for connecting aws kinesis. When I'm trying to connect with FlinkKinesisConsumer it's throwing an error like Classnotfound exception.
import configs.AWSConfigConstants;
import…
Sanjaybabu
- 1
- 1
0
votes
1 answer
Should Flink State be used for large and medium term storage?
Before starting, what I mean by large is GBs and medium-term storage is hours. We have a Flink running on AWS Kinesis Data Analytics for Flink Applications (KDA), which uses RockDB State Backend by default. Each KPU in KDA (kind of like task…
George
- 1
- 1
0
votes
1 answer
Serialization of java.util.map and custom pojo in Flink
I am trying to use a pojo which does not implement Serializable from an external library in the kinesis flink application .The serialization is failing while using it inside flatMap function.
Pojo
public class ExecutionRecord {
private…
Yash Jain
- 1
- 1
0
votes
1 answer
Using Python Processors in Java Flink Application
I have a use case where I want to implement an AWS Kinesis Data Application with Flink in Java. It will listen to multiple Kinesis streams via the Data Streams API. However, the analysis of those streams will be done in Python (since our data…
Alex
- 349
- 1
- 10
0
votes
1 answer
AWS Kinesis Multipart Upload to an Amazon S3 bucket
I want to do a multipart upload from Kinesis into my Amazon S3 bucket. As Amazon S3 is a file system, for every entry it will create a file under the given bucket name.
My Amazon S3 feeds the AWS glue jobs as it triggers the Lambda function as soon…
addytheriot
- 31
- 1
- 6
0
votes
1 answer
Flink kafka consumer lagging behind
I'm consuming stream from Kafka source for my flink job reading from 50 topics at once like this:
FlinkKafkaConsumer kafkaConsumer = new FlinkKafkaConsumer(
Pattern.compile("TOPIC_NAME[1-50].stream"), //…
Muhammad Bilal Javed
- 41
- 7
0
votes
1 answer
Apache Flink metric to count late elements
I'd like to measure how many events arrive within allowed lateness grouped by particular feature of the event. We assume particular type of events have way more late arrivals and would like to verify this.
The place to make the measurement I thought…
sumek
- 26,495
- 13
- 56
- 75
0
votes
0 answers
Why Kinesis Data Analytics for Flink drops state when scaled up or scaled down?
We are using Kinesis for Apache Flink to analyze various visitor events from multiple sources. In one of the operators, we are using a MapSate for cumulative metrics calculation. Flink application was auto-scaled 4 times during one-week execution.…
Yuriy Zanichkovskyy
- 1,689
- 11
- 16