Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-analytics]

Amazon Kinesis Data Analytics is the way to analyze streaming data, gain actionable insights, and respond in real time. SQL users can query streaming data or build entire streaming applications using templates and an interactive SQL editor. Java developers can build streaming applications using open source Java libraries and AWS integrations to transform and analyze data in real-time.

133 questions
0
votes
3 answers

Flink Optimization Larger Aggregations or More Tasks

For Apache Flink aggregations is it better to have an aggregation with complex state or to have smaller aggregations but more tasks. For example, if I have a data stream on users watching videos over a web interface. I want aggregations for: How…
Thor
  • 600
  • 1
  • 6
  • 17
0
votes
1 answer

How to deploy Kinesis Data Analitics without data loss

We have a complex IoT system in which an AWS Kinesis Flink application collects data, processes it, and forwards new data packages to another system. We collect a lot of small data, such as data from sensors (around 100 per sensor per day), and…
ForestG
  • 17,538
  • 14
  • 52
  • 86
0
votes
0 answers

logically Isolating data in apache Flink

We are processing around 30 million of records per day using a apache flink job, this flink job filters the data data from source kinesis stream and push the filtered data to the respective kinesis streams which are on some other AWS account, here…
Yogesh Katkar
  • 133
  • 10
0
votes
1 answer

Why does Flink ValueState.value() sometimes erroneously return null?

I am encountering an error in my Flink app where calling myValueState.value(), inside a KeyedProcessFunction, sometimes returns null despite the fact that the logic in the code should guarantee that the object returned by .value() is not null. These…
r_g_s_
  • 224
  • 1
  • 8
0
votes
2 answers

AWS Kinesis: Concurrency Capabilities

I have been struggling with finding accurate information for concurrency capabilities of AWS Kinesis. So far, my understanding of intake concurrency and output concurrency of AWS Kinesis differ from one another (DataStream, Firehose, and…
Shin Yoo
  • 1
0
votes
0 answers

AWS KDA open flink dashbaord without presigned url

can we create a permanent dashboard url for KDA flink application ?. we have a requirement to use it without the createpresignedurl permission.
Fryder
  • 413
  • 2
  • 7
  • 21
0
votes
0 answers

multiple consumers reading from same shard in AWS Kinesis

When multiple consumers are reading from same shard in AWS Kinesis, when a consumer reads a message, is that message deleted and not accessible for other consumers? I have multiple consumers for a kinesis shard, when the second consumer is making…
abhinav
  • 527
  • 3
  • 11
  • 24
0
votes
0 answers

External Monitoring for Flink applications running on AWS

We have a Flink application that runs on AWS Kinesis Data Analytics. Currently, we use CloudWatch to monitor the application's health and performance. Since Flink allows reporting metrics to an external system, we would like to extend our monitoring…
aveek
  • 188
  • 6
0
votes
0 answers

Flink Serialization: Scala Case Class Treated as Generic Type

I've noticed that Kryo is using up a lot of application CPU and turned off genericTypes in order to debug. Based on the documentation, it seems like Scala case classes should be serialized by a Flink Serializer, not Kryo, but I'm getting this…
kellanburket
  • 12,250
  • 3
  • 46
  • 73
0
votes
1 answer

Flink/Kinesis Analytics: Even Key Groups Across SubTasks

I have a simple Flink/Kinesis Analytics application with two task slots: Source -> Transform, Repartition -> Sink. My application has 32 KPUs with a parallelism of 1, reading from a Kinesis Stream with 60 shards. After a transformation stage I key…
kellanburket
  • 12,250
  • 3
  • 46
  • 73
0
votes
0 answers

Can KinesisStreamsSink retain the partition order from the original stream?

I'm currently using a Kinesis data stream as a command queue for a high number of instances. The original stream is using the instance-id as the partition key. The goal is to get the ordered events to a worker pool. Each worker pool is capable of…
peterulb
  • 2,869
  • 13
  • 20
0
votes
0 answers

I see half the parallelisms in Cloudwatch KDA

I have a KDA application that is configured to have 2 KeyedProcess operators with 96 parallels for each operator. So I should see 192 entries with 'Task: KeyedProcess' and any kind of metric (in my case 'numRecordsInPerSecond'). But I only see 96 in…
Marco
  • 41
  • 5
0
votes
1 answer

Apache Flink issue with JOINS on Kinesis Streams Rowtime attributes must not be in the input rows of a regular join

i am attempting a simple exercise i have Two kinesis data stream order-stream shipment-stream SQL 1 Orders %flink.ssql CREATE TABLE orders ( orderid VARCHAR(6), orders VARCHAR, ts TIMESTAMP(3), WATERMARK FOR ts AS ts - INTERVAL…
Soumil Nitin Shah
  • 634
  • 2
  • 7
  • 18
0
votes
0 answers

Flink Kinesis Sink Decoding Issue

I have Flink job running in AWS Kinesis Analytics that does the following. 1 - I have Table on a Kinesis Stream - Called MainEvents. 2 - I have a Sink Table that is pointing to Kinesis Stream - Called perMinute. The perMinute is populated using the…
Up_One
  • 5,213
  • 3
  • 33
  • 65
0
votes
1 answer

Are there any Kinesis Connectors for Python DataStream API Flink 1.13 version?

I am trying to build a streaming application using Kinesis Data Analytics with Flink 1.13 version in Python. The source for the application is the Kinesis data stream. but I can see that the kinesis connector FlinkKinesisConsumer is not available in…
SIVA KRISHNA
  • 1
1 2 3
…
8 9