Stack Overflow
Stack Overflow

Questions tagged [flink-sql]

Apache Flink features two relational APIs, SQL and Table API, as unified APIs for stream and batch processing.

Apache Flink features two relational APIs:

  1. SQL (via Apache Calcite)
  2. Table API, a language-integrated query (LINQ) interface

Both APIs are unified APIs for stream and batch processing. This means that the a query returns the same result regardless whether it is applied on a static data set or a data stream. SQL queries are parsed and optimized by Apache Calcite (Table API queries are optimized by Calcite).

Both APIs are tightly integrated with Flink's DataStream and DataSet APIs.

667 questions
3
votes
0 answers

Flink TableAPI: Failed to Fetch

I am playing around with the Flink tableAPI right now. I created a table, and want to see what's in the table. I thought if I use a SELECT statement with .collect(), I could get a iterator which I could use to iterate and see what's inside the…
Liu
  • 125
  • 7
3
votes
1 answer

Failed to connect to Confluent Platform Schema Registry - Apache Flink SQL Confluent Avro Format

I am using Confluent managed Kafka cluster, Schema Registry service and trying to process Debezium messages in a Flink job. The job is configured to use Table & SQL Connectors and Confluent Avro Format. However the job is not able to connect to…
Maksym Kovalenko
  • 31
  • 1
3
votes
2 answers

Flink - Convert Avro datastream to table

I have messages in Avro format in Kafka. These have to converted to table and selected using SQL, then converted to stream and finally sink. There are multiple Kafka topics with different Avro schemas, hence dynamic tables are required. Here is the…
warrior107
  • 709
  • 1
  • 9
  • 25
3
votes
1 answer

Flink SQL: source table is too big to fit into memory

I'm relatively new to Flink and today I encountered a problem while using Flink SQL on Flink 1.11.3 session cluster. Problem I registered a source table which uses jdbc postgres driver. I am trying to move some data from this online DB to AWS S3 in…
Patrick
  • 117
  • 4
3
votes
1 answer

Flink Cluster startup Error : Could not resolve ResourceManager address akka

Need help with following error as I dont seem to find what is actual issue. I am trying to run flink cluster on docker-desktop in win10 professional. Dockerfile: FROM SOME-LOCAL-REGISTERY-URL/flink:1.11 ADD build/libs/demoapp-service-all.jar…
ParagM
  • 63
  • 1
  • 7
3
votes
1 answer

Flink Table API: GROUP BY in SQL Execution throws org.apache.flink.table.api.TableException

I have this very simplified use case: I want to use Apache Flink (1.11) to read data from a Kafka topic (let's call it source_topic), count an attribute in it (called b) and write the result into another Kafka topic (result_topic). I have the…
Irina S.
  • 133
  • 8
3
votes
1 answer

"Rowtime attributes must not be in the input rows of a regular join" despite using interval join, but only with event timestamp

Example code: from pyflink.table import EnvironmentSettings, StreamTableEnvironment env_settings = ( EnvironmentSettings.new_instance().in_streaming_mode().use_blink_planner().build() ) table_env =…
Alex Hall
  • 34,833
  • 5
  • 57
  • 89
3
votes
1 answer

Create FlinkSQL UDF with generic return type

I would like to define function MAX_BY that takes value of type T and ordering parameter of type Number and returns max element from window according to ordering (of type T). I've tried public class MaxBy extends AggregateFunction
bottaio
  • 4,963
  • 3
  • 19
  • 43
3
votes
3 answers

why do we have flink-streaming-java and flink-streaming-scala modules in flink source code

In Fink source, there are flink-stream-java and flink-stream-scala modules. Why do we need two modules for flink streaming?…
yuyang
  • 1,511
  • 2
  • 15
  • 40
3
votes
1 answer

Apache Flink - enable join ordering

I have noticed that Apache Flink does not optimise the order in which the tables are joined. At the moment, it keeps the user-specified join order (basically, it takes the the query literally). I suppose that Apache Calcite can optimise the order…
Salvatore Rapisarda
  • 97
  • 6
3
votes
1 answer

Flink Table/SQL API: modify rowtime attribute after session window aggregation

I want to use Session window aggregation and then run Tumble window aggregation on top of the produced result in Table API/Flink SQL. Is it possible to modify rowtime attribute after first session aggregation to have it equal a .rowtime of the last…
rwozniak
  • 43
  • 4
3
votes
1 answer

Difference between DataStream and Table API in Apache Flink

I am new to Apache Flink and want to understand use case between DataStream and Table API. Please help me understand when to choose Table API over DataStream API. As per my understanding, things which can be done using Table API can also be done…
Shailendra
  • 347
  • 6
  • 21
3
votes
0 answers

Create composite type in flink table

I am trying to write a user-defined scalar function in Flink which takes in multiple expressions (arbitrary number of expressions) and combine that into a single expression. Coming from Spark world, I could achieve this by using struct which returns…
Rakesh Chalasani
  • 313
  • 1
  • 7
3
votes
1 answer

how to stream a json using flink?

i 'm actually working on a stream, receiving a bunch of strings and need to make a count of all the strings. the sums is aggragated, that mean for the second record the sum was added to the day before the output must be some json file looking…
TheEliteOne
  • 105
  • 1
  • 11
3
votes
1 answer

Apache Flink: Best way to architect many-to-one join on dynamic table?

I have a resource with frequently added/updated rows called Orders, and a resource called UserProfile with less frequent (but often important) updates. I want to perform a continuous join query on these two resources (currently stored logically as…
Austin York
  • 808
  • 2
  • 10
  • 24
1
2
3
…
44 45