Stack Overflow
Stack Overflow

Questions tagged [flink-sql]

Apache Flink features two relational APIs, SQL and Table API, as unified APIs for stream and batch processing.

Apache Flink features two relational APIs:

  1. SQL (via Apache Calcite)
  2. Table API, a language-integrated query (LINQ) interface

Both APIs are unified APIs for stream and batch processing. This means that the a query returns the same result regardless whether it is applied on a static data set or a data stream. SQL queries are parsed and optimized by Apache Calcite (Table API queries are optimized by Calcite).

Both APIs are tightly integrated with Flink's DataStream and DataSet APIs.

667 questions
0
votes
0 answers

How to run TableAPI and DataStreamAPI in parallel to use Python defined functions

I have a job with DataStream API, which is running fine, but I need to use the DataStream resulting from a computation and pass it to the TableAPI to call a register python function and then pass the result back to a new DataStream to…
Alter
  • 903
  • 1
  • 11
  • 27
0
votes
1 answer

Flink- multi event dependency sql query on dataStream

Not getting expected behavior, my flink application getting live event and my trigger condition is depend on two event ABC and XYZ. when both event reach then trigger the notification. application is using StreamTableEnviornment here is the sql…
Ashutosh
  • 33
  • 8
0
votes
1 answer

Joining continuous queries in Flink SQL

I'm trying to join two continuous queries, but keep running into the following error: Rowtime attributes must not be in the input rows of a regular join. As a workaround you can cast the time attributes of input tables to TIMESTAMP before.\nPlease…
Daniel Lenz
  • 3,334
  • 17
  • 36
0
votes
2 answers

Flink savepoint with local execution environment (like standalone application)

How can I implement flink savepoint with standalone application (local execution env or mini cluster). I configured savepoint directory in flink-config.yaml file but not sure how to take the savepoint before shutdown the application and how to…
Ashutosh
  • 33
  • 8
0
votes
1 answer

Flink forward files from List filePaths

We have a list of filepaths from a DB table with a timestamp on when it is created. Trying to figureout how we can use the filepath list from db to forward only those files from nfs to kafka sink. Right now I am using customized version of…
VSK
  • 359
  • 2
  • 5
  • 20
0
votes
1 answer

Apache Flink SQL InvalidProgramException: Selected sort key is not a sortable type

This is my Flink SQL SELECT t.reportCode FROM query_record_info as t LEFT JOIN credit_report_head as c ON t.reportCode = c.reportCode when i run it,i got a wrong Exception in thread "main" org.apache.flink.api.common.InvalidProgramException:…
herion
  • 3
  • 2
0
votes
2 answers

FLINK - will SQL window flush the element on regular interval for processing

I am confused if TUMBLE window will get calculated on regular interval and emit the elements for processing. example I have a query that is expected to work on interval 10 second. select id, key from eventTable GROUP BY TUMBLE(rowTime, INTERVAL…
Ashutosh
  • 33
  • 8
0
votes
1 answer

FLINK- how to process logic on sql query result

My requirement is to process or build some logic around the result of sql query in flink. For simplicity lets say I have two sql query they are running on different window size and one event stream. My question is a) how I will know for which query…
Ashutosh
  • 33
  • 8
0
votes
1 answer

How does TM recovery handle past broadcasted data

In the context of HA of TaskManagers(TM), when a TM goes down a new one will be restored from latest checkpoint of faulted by the JobManager(JM). Say we have 3 TMs (tm1, tm2, & tm3) At a give time t where everyone's checkpoint(cp) is at cp1. All TMs…
ardhani
  • 303
  • 1
  • 11
0
votes
1 answer

FLINK- Load historical data and maintain window of 30 days

My requirement is to hold 30 days data into stream to given any day for processing. so first day when FLINK application will start, it will fetch 30 days data from database and will merge to current stream data. My challenge is - manage 30 days data…
Ashutosh
  • 33
  • 8
0
votes
1 answer

What backend does Flink Table API use ? Does it require any relational DB?

I'm fairly new to Flink and trying to understand appropriate use cases where Stream API/ Table API can be used. As part of it trying to understand like Stream API, does Table API has the flexibility to choose the type of state backend it can…
ardhani
  • 303
  • 1
  • 11
0
votes
1 answer

Flink combination of windowByTime and triggerByCount

source.keyBy(0) .window(TumblingEventTimeWindows.of(Time.seconds(5))) .trigger(PurgingTrigger.of(CountTrigger.of[TimeWindow](2))) .process(new TestFun()) Explanation: Let's say I have 3 events[E1, E2, E3], which should be trigger by…
Tulasi
  • 79
  • 1
  • 9
0
votes
1 answer

Stream kinesis Analytics ETL Flink - skip records before and after a delay

EDITED: I have a requirement to skip records that are created before 10s and 20s after if a gap in incoming data occurs. (A gap is said to occur when the event-time1 - event-time2 > 3 seconds) the resulting data is used to calculate average or…
Ajmal M Sali
  • 598
  • 6
  • 14
0
votes
1 answer

Is it possible to run a simple job in Job Manager in Flink?

I have written a Flink job that reads data from Kafka and writes to hdfs file in ORC format for HIVE(uses 20 executors). I need to run a simple job that inserts partition to the HIVE table every hour. Is it possible to run this simple partition…
patel akash
  • 75
  • 8
0
votes
0 answers

Flink 1.10 match recognize Order BY giving error . #apache-flink

I am using match recognize query to my stream and order by (event_time) where event_time field is of type timestamp. I have added assignTimestampsAndWatermarks() to the field in stream and then create table followed by the query but i get the error…
IraSantuManu lifeblog
  • 1
  • 2
1 2 3
…
44 45