Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-analytics]

Amazon Kinesis Data Analytics is the way to analyze streaming data, gain actionable insights, and respond in real time. SQL users can query streaming data or build entire streaming applications using templates and an interactive SQL editor. Java developers can build streaming applications using open source Java libraries and AWS integrations to transform and analyze data in real-time.

133 questions
1
vote
1 answer

Kinesis Data Analytics (Flink) - how do I configure environment variables?

we are currently running a flink cluster in a standalone mode on Kubernetes. We have wanted to explore whether we could migrate over to managed flink on AWS (KDA). But I don't seem to find any documentation or indication that it is possible to…
me_L_coding
  • 169
  • 10
1
vote
0 answers

pyFlink submit job with multiple external connector jars on Amazon Kinesis

I am follwing this guide to create an Amazon Kinesis Analytics Application with pyflink, and my application requires more than 1 external connector jarfile. When it comes to the jarfile uploading section, it seems I can only upload 1 jarfile, how…
chris
  • 11
  • 2
1
vote
0 answers

Add VPC Configuration in Kinesis Data Analytic App via aws cdk

I am trying to create AWS Kinesis Data Analytic(KDA) App via cdk. I need to put this KDA app inside a VPC but I am not able to find any cloudformation(cdk) support to add it. I can see vpc configiration api is exposed by aws kinesis sdk but it is…
priyadhingra19
  • 333
  • 4
  • 15
1
vote
0 answers

AWS Kinesis Firehose to DynamoDB through AWS Lambda

I am creating an architecture where Producers sends data to AWS Kinesis Firehose. I need to store the data into DynamoDB through AWS Kinesis. According to documentation, DynamoDB is not supported as one of the destinations. Is it possible to use…
Ronnie
  • 483
  • 1
  • 5
  • 18
1
vote
1 answer

Getting Error when pre-processing data from Kinesis with Lambda

I have a use case where I have to filter incoming data from Kinesis Firehose based on the type of the event. I should write only certain events to S3 and ignore the rest of the events. I am using lambda to filter the records. I am using following…
Harish J
  • 146
  • 1
  • 3
  • 12
1
vote
2 answers

KDA metrics in cloudwatch different from Flink metrics

I have a Flink application deployed on AWS Kinesis Data Analytics. My current setting is: Parallelism=128 Parallelism per KPU=4 The issue I have is, there is a big difference between the counts shown on the Flink web UI vs the Count shown in…
Vinod Mohanan
  • 3,729
  • 2
  • 17
  • 25
1
vote
1 answer

Kinesis Analytics SQL query to narrow down the sensors that are not sending data

Context: We use Kinesis analytics to process our sensor data and find anomalies in the sensor data. Goal: We need to identify the sensors that didn’t send the data for the past X minutes. The following methods have been tried with Kinesis analytics…
Lakshman Diwaakar
  • 7,207
  • 6
  • 47
  • 81
1
vote
1 answer

Zeppelin Python Flink cannot print to console

I'm using Kinesis Data Analytics Studio which provides a Zeppelin environment. Very simple code: %flink.pyflink from pyflink.common.serialization import JsonRowDeserializationSchema from pyflink.common.typeinfo import Types from pyflink.datastream…
dz902
  • 4,782
  • 38
  • 41
1
vote
1 answer

Order of records from Kinesis Shard in Flink processing

I'm having a problem understanding how to preserve the order of events when consuming records from a Kinesis stream with Flink. Our setup looks like this: Kinesis stream with 8 shards Sharding key is the userId of the user that produced the…
shinlang
  • 37
  • 1
  • 7
1
vote
1 answer

AWS Kinesis Analytics SQL to transform rows to a list

I have a Destination Stream that looks as follows for example : Company_ID Product User_Tag Count 123 P1 T1 15 123 P1 T2 10 123 P1 T3 …
Ram K
  • 1,746
  • 2
  • 14
  • 23
1
vote
1 answer

How to configure graphite metrics reporter for kinesis data analytics application

I am running a Flink application as part of the AWS Kinesis Data Analytics service. Flink has built in support for metrics and I have a simple counter setup that I can see is working, it is available in the flink dashboard. Now, I want to configure…
HenriR
  • 11
  • 2
1
vote
0 answers

Apache Flink : Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Could not serialize inputs

I have a Flink-1.13 program that reads data from a Kinesis stream containing records with different schemas. My program iterates over all the possible schemas contained into the stream, filters a main dataframe and write GenericRecord records into…
nach0
  • 379
  • 1
  • 3
  • 14
1
vote
2 answers

Unable to create a source for reading table error when trying to query a glue table

I'm new to kinesis analytics studio using apache flink, basically I have a data stream with hundreds of records going through it, and I want to make some real-time basic analysis, so I went through this tutorial and created a table in glue using…
JCLOUD
  • 61
  • 5
1
vote
1 answer

Authenticating with Google Cloud from Apache Beam application via code

I'm trying to run an Apache Beam application in Kinesis Data Analytics which uses Apache Flink as the runtime. The pipeline uses the PubsubIO connector. I'm trying to authenticate with Google Cloud using code, since Kinesis Data Analytics does not…
Gayan Weerakutti
  • 11,904
  • 2
  • 71
  • 68
1
vote
0 answers

Putting all records in an array column in Amazon Kinesis Analytics SQL?

I want to use Analytics SQL to combine all windowed values of column in the input stream into an array in the output stream. Say I have a data coming in that looks like…
stumpbeard
  • 61
  • 8
1 2
3
…
8 9