Stack Overflow
Stack Overflow

Questions tagged [flink-sql]

Apache Flink features two relational APIs, SQL and Table API, as unified APIs for stream and batch processing.

Apache Flink features two relational APIs:

  1. SQL (via Apache Calcite)
  2. Table API, a language-integrated query (LINQ) interface

Both APIs are unified APIs for stream and batch processing. This means that the a query returns the same result regardless whether it is applied on a static data set or a data stream. SQL queries are parsed and optimized by Apache Calcite (Table API queries are optimized by Calcite).

Both APIs are tightly integrated with Flink's DataStream and DataSet APIs.

667 questions
1
vote
0 answers

Using Python Functions in a Java Flink Job - 1.15

Is there any way to use a python function (Aggregate, Map etc.) within a Java Flink Job? I do not want to exploit SQL API. I wonder if only DataStream API can handle such functionality? Without this syntax: tableEnv.executeSql("CREATE TEMPORARY…
Metehan Yıldırım
  • 352
  • 2
  • 10
1
vote
1 answer

How to connect to MinIO filesystem in Flink sql client?

I'm trying to build data pipeline built with Flink and MinIO as the storage, currently I can sink the data to MinIO bucket success, but when I try to create a table WITH the minio file, it always encountered the Connection Refused error: Flink SQL>…
Liping Huang
  • 4,378
  • 4
  • 29
  • 46
1
vote
0 answers

Flink Nested Json parsing parsing with complex schema

We have a requirement of parsing a very complex json (size around 25 kb per event) event with a predefined schema (nested schema, with multiple schema files ) and create a temporary table and from temp table we have to apply some case statement…
Soumen C
  • 136
  • 3
1
vote
2 answers

How to connect to Flink SQL Client from NodeJS?

I'm trying to use Apache Flink's Table concept in one of my projects to combine data from multiple sources in real-time. Unfortunately, all of my team members are Node.JS developers. So, I'm looking for possible ways to connect to Flink from NodeJS…
Stranger
  • 10,332
  • 18
  • 78
  • 115
1
vote
1 answer

Obtain Source (Sink) out of Source (Sink) and f:A->B (f:B->A)

In order to build sources & sinks on top of existing ones (as opposed to do it from scratch or with more boilerplate), I'd like to: Append a conversion function f:A->B to a Source in order to get a Source (corresponds to map in FP…
salvalcantara
  • 420
  • 4
  • 11
1
vote
0 answers

Unexpected type: BINARY

I am trying to read parquet files via the Flink table, and it throws the error when I select one of the timestamps. My parquet table is something like this. I create a table with this SQL : CREATE TABLE MyDummyTable ( `id` INT, …
None
  • 330
  • 2
  • 16
1
vote
1 answer

Flink SQL deduplication state management

I have a use case to deduplicate the data using Table API (while streaming the data from one source to another sink). This documentation looks very clear for such use case. But what I don't understand is that, how the state management works in this…
Invisible
  • 179
  • 1
  • 8
1
vote
0 answers

Flink Table API SQL throws "Cannot read strings longer than 2147483639 bytes"

I am a newbie to Flink/Avro. I am trying Flink 1.14.4 Table API to read the Avro format from Kafka (not the confluent one). I am not able to read any of the message. I am getting the following exception, Caused by:…
Invisible
  • 179
  • 1
  • 8
1
vote
0 answers

How to convert Table containing TIMESTAMP_LTZ into DataStream in PyFlink 1.15.0?

I have a source table using a Kinesis connector reading events from AWS EventBridge using PyFlink 1.15.0. An example of the sorts of data that are in this stream is here. Note that the stream of data contains many different types of events, where…
John
  • 10,837
  • 17
  • 78
  • 141
1
vote
2 answers

Unable to resolve Flink table API dependency

I am using IntelliJ IDE for development. I am using below mentioned POM file for dependency management