Stack Overflow
Stack Overflow

Questions tagged [flink-sql]

Apache Flink features two relational APIs, SQL and Table API, as unified APIs for stream and batch processing.

Apache Flink features two relational APIs:

  1. SQL (via Apache Calcite)
  2. Table API, a language-integrated query (LINQ) interface

Both APIs are unified APIs for stream and batch processing. This means that the a query returns the same result regardless whether it is applied on a static data set or a data stream. SQL queries are parsed and optimized by Apache Calcite (Table API queries are optimized by Calcite).

Both APIs are tightly integrated with Flink's DataStream and DataSet APIs.

667 questions
2
votes
0 answers

Query on Flink SQL primary key for nested field

I have a Kafka topic named employee which uses confluent avro schema and will emit the payload as below: { "employee": { "id": "123456", "name": "sampleName" } } I am using Apache Flink's upsert-kafka connector to consume the events from the above…
JDP
  • 35
  • 6
2
votes
0 answers

Flink Table API : Flink Dynamic table produces incorrect intermediate CDC values

I am using flink table API to calculate few aggregations. I have stream of data coming from Kafka which is transformed to stream of rows. Using this rows I am creating dynamic table. Ex: consider below three records, primary key is…
Aishwarya
  • 21
  • 1
2
votes
1 answer

Creating a Tumbling Window for a Flink table

I'm trying to create a tubmbling window for a Flink table in Java and query data for that window Table table = tEnv.sqlQuery(query.getQuery()); // create a new column 'EventTime' of type Timestamp from 'EventTimetamp' which is a string table =…
shepherd
  • 33
  • 3
2
votes
4 answers

Why flink 1.15.2 showing No Watermark (Watermarks are only available if EventTime is used)

In my create table ddl, i have set watermark on column and doing simple count(distinct userId) on a tumble window of 1 min, but stil not getting any data, same simple job is working fine in 1.13 CREATE TABLE test ( …
user9068199
  • 55
  • 4
2
votes
0 answers

How to use Groovy and Spock to test Apach Flink Job?

I have a Flink job that reads data from Kafka into a table which is emitted into a DataStream on which I apply a filter function and then convert the data stream back to a table which writes data back to Kafka. I want to test the functionality of…
prgx
  • 41
  • 3
2
votes
0 answers

Use FlinkSQL to read Kafka's data to the console

package FlinkAPI; import org.apache.flink.api.common.functions.FilterFunction; import org.apache.flink.api.common.functions.MapFunction; import org.apache.flink.api.java.tuple.Tuple2; import…
SOBER
  • 21
  • 1
2
votes
1 answer

How to collect side output from rich sink function in Apache Flink?

I am trying below scenario in Flink Flink consume data from kafka topic and validate against avro schema Converting the data into JSON payload in process function after some enrichments on the data After enrichment of data of it should be written…
user13906258
  • 161
  • 1
  • 13
2
votes
2 answers

Flink-SQL: Extract values from nested objects

i'm using Flink SQL and the following scheme shows my source data (belonging to some Twitter data): CREATE TABLE `twitter_raw` ( `entities` ROW( `hashtags` ROW( `text` STRING, `indices` INT ARRAY ) ARRAY, …
JanOels
  • 111
  • 11
2
votes
0 answers

How to use Flink SQL to create a table with the avro schema which has a complicated array type

I try to use FlinkSQL to create a table with Avro format, but the avro schema is a little bit complicated which is like this: { "type":"record", "name":"Record", "namespace":"example.avro", "fields":[ { …
chenyl
  • 21
  • 1
2
votes
1 answer

Apache Flink - Matching multiple patterns

We have a requirement where multiple events will be matched through the given pattern. Pattern 1 followed by within 10 mins Pattern 2 followed by within 10 mins Pattern 3. Pattern 1: 10 failed logins from the same username and same device within 10…
JDForLife
  • 91
  • 2
  • 10
2
votes
0 answers

Flink Sql not converting RAW('org.apache.avro.util.Utf8', '...') to String

I'm reading from a kafka stream, creating a Table environment and calculating an average and writing the data back to kafka [SIMPLECUSTOMER]. This worked in Flink 1.12.5. I'm using Flink 1.13.2 and Flink 1.14.0 customerId is read as…
Mahendran Ponnusamy
  • 31
  • 4
2
votes
1 answer

How to extract nested JSON object from kafka in flink table?

Here is a json format message in kafka. { "@timestamp": "2021-08-14T11:11:00.301Z", "@metadata": { "beat": "filebeat", "type": "_doc", "version": "7.14.0" }, "message": { "k1": "v1", "k2":…
king
  • 300
  • 1
  • 13
2
votes
1 answer

Apache Flink delay processing of certain events

I have a requirement to delay processing of some of the events. eg. I have three events (published on Kafka): A (id: 1, retryAt: now) B (id: 2, retryAt: 10 minutes later) C (id: 3, retryAt: now) I need to process record A and C immediately while…
Hamid
  • 717
  • 7
  • 15
2
votes
1 answer

Sink flink DataStream using jdbc connector to mysql sink with overwrite

My use case is Get Data from AWS Kinesis Data stream and filter/map using flink data stream api Use StreamTable Environment to group and aggregate data Use SQLTableEnvironment to write to mysql using JDBC Connector I am able to write my datastream…
Mujahid
  • 114
  • 1
  • 3
  • 14
2
votes
0 answers

Multiple select queries in single job on flink table API

If I want to run two different select queries on a flink table created from the dataStream, the blink-planner runs them as two different jobs. Is there a way to combine them and run as a single job ? Example code : StreamExecutionEnvironment env =…
voidMainReturn
  • 3,339
  • 6
  • 38
  • 66
1 2
3
…
44 45