Stack Overflow
Stack Overflow

Questions tagged [flink-streaming]

Apache Flink is an open source platform for scalable batch and stream data processing. Flink supports batch and streaming analytics, in one system. Analytical programs can be written in concise and elegant APIs in Java and Scala.

Flink's streaming API provides rich semantics, including processing- and event-time windows, as well as stateful UDFs. Flink streaming uses a light-weight fault-tolerance mechanism with exactly-once processing guarantees.

Learn more about Apache Flink at the project website: https://flink.apache.org/

3185 questions
0
votes
1 answer

How to setup apache Flink local execution environment with multi-threading?

I would like to test apache flink partitioning. For that I created a simple program in Scala. The idea is, a source generates 0 and 1, then key by them, and print the thread id and subtask id in the processor. I expect that 0 and 1 are processed in…
Xiang Zhang
  • 2,831
  • 20
  • 40
0
votes
0 answers

we started getting the below error after we bumped the flink version from 1.15 to 1.17

We have upgraded to flink 1.17 from 1.15. After upgrading the job, we are facing exceptions which are causing the job to fail. We are using fixed-delay restart strategy and the job is deployed on kubernetes…
Shiv Desai
  • 1
0
votes
1 answer

AWS SQS Sink in Flink

I want to add Aws SQS sink in Flink. Wanted to know if anybody has already tried the same or if there is any known challenges with this integration? Any code references or documentation would be helpful.
priyadhingra19
  • 333
  • 4
  • 15
0
votes
1 answer

Flink- force Checkpoint

Currently, flink application is configured and implemented to create avro files on every checkpoint. Is is possible to force the flink application to create avro file on-demand, instead of configurable time interval. Is there any REST APIs or any…
Hareesh
  • 41
  • 4
0
votes
0 answers

Flink - Changing deployment parameters that affects on going windows

Im using SlidingProcessingTimeWindows (Flink-1.18) with configureable parameters for size and slide. Further more, I have another configureable parameter that decides if should I run a special calculation (implemented in "getResult" and in "Add"…
Orel Yamin
  • 15
  • 4
0
votes
0 answers

Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec (create-build-target-link) on project flink-dist_2.12

I am trying to run ./mvnw clean package -DskipTests command on Flink-1.16.0 source code and build is getting failed on flink-dist module with following error. Please let me know if there is any way to fix it. I am using JDK 11.0.19 with Maven :…
priyadhingra19
  • 333
  • 4
  • 15
0
votes
0 answers

Flink stream data from S3, directories partitioned by time

I want to use S3 as a data source where the objects are partitioned by hour in the following format: s3://test-data//type/. I discovered I can use FileSource with the monitorContinuously function, but I’m trying to find a way to ensure that at any…
Developer
  • 299
  • 1
  • 2
  • 11
0
votes
0 answers

Flink SQL lookup table left join dynamic table

I have two tables - Customers and Orders. My requirement is to count the orders of every customer for each minute. I want to use Flink SQL to implement this task. My approach is as follows: CREATE TEMPORARY TABLE customers ( id INT, name…
Kerie
  • 121
  • 5
0
votes
0 answers

await method on TableResult is not working when job is submitted via Session Mode using Apache Flink Operator

await method on TableResult is not working when job is submitted via Session Mode using Apache Flink Operator by creating FlinkSessionJob resource in kubernetes. The same code is working when the job is deployed using the application mode. Here is…
Vinay Cheguri
  • 55
  • 7
0
votes
0 answers

How to achieve append only results when inserting data to output table after a condition match on input data - flink streaming

Flink Version - 1.16.1 Use case -> Input - I have some data coming in from a kafka topic(not compacted). Say, This data has information about some user activity on a computer. I have a file that has a list of blacklisted users. Users can be added…
Neha
  • 225
  • 1
  • 5
  • 12
0
votes
0 answers

'CoderAdapterIml' object has no attribute 'encode' error with Custom Serializer pyflink

I am trying to create a custom window, and right now I am facing problems with serialization. I have this as my current serializer class CustomSerializer(TypeSerializer): def serialize(self, element, stream) -> None: bytes_data =…
Qwetroman
  • 11
  • 3
0
votes
0 answers

PyFlink window aggregation not triggering

I have a problem that my window aggregation accumulates all results, but do not return it, and my result stream is empty I suspect it has something to do with windows triggering, but cannot figure out…
Qwetroman
  • 11
  • 3
0
votes
1 answer

Using HDFS with Apache Flink RocksDBStateBackend on Windows: Step-by-step tutorial

How to configure Apache Flink RocksDBStateBackend setup locally. System configuration: Win10 + Intellij Idea 2021.3 + Apache Flink 1.9 Now everything works locally with the Memory State Back end. As I understand it, when starting Idea raises a local…
Igor
  • 3
  • 1
0
votes
2 answers

Getting Async Task Checkpoint failed error with Flink Presto 1.14.0 library

I am trying to use Flink S3 checkpointing with Flink s3 presto library. Following is my s3 related flink configs : fs.allowed-fallback-filesystems: s3p state.backend: filesystem state.checkpoints.dir:…
priyadhingra19
  • 333
  • 4
  • 15
0
votes
0 answers

Flink job submission to Multi-VM Flink Cluster fails with JobSubmissionException, CompletionException and Connection Refused error

I can launch a Flink cluster (version 1.17.x) on my laptop with 1 Job Manager and 3 Task Managers. The cluster starts, jobs can be submitted correctly on the localhost (my laptop). Next step is to launch this cluster on 4 VMs - 1 Master VM (for the…
user1831595
  • 1
  • 2
1 2 3
…
99
100