Stack Overflow
Stack Overflow

Questions tagged [flink-batch]

158 questions
1
vote
1 answer

How to use Apache Flink CEP SQL for getting events from already matched pattern?

My requirement is to generate trigger based on 2 events(EVT_A & EVT_B independent of order). Here is expectation 1. EVT_A arrived. --> No action 2. EVT_B arrived --> Should Trigger 3. EVT_B arrived --> should Trigger since A was received…
ParagM
  • 63
  • 1
  • 7
1
vote
1 answer

Flink - multiple instances of flink application deployment on kubernetes

I need help on Flink application deployment on K8 we have 3 source that will send trigger condition as in form of SQL queries. Total queries ~3-6k and effectively a heavy load on flink instance. I try to execute but it was very slow and takes lot of…
Ashutosh
  • 33
  • 8
1
vote
1 answer

readCsvFile error: How to specify implicit value for evidence parameter of type

I am working in IntelliJ to create and test a machine learning model that will classify incoming data from a stream. I am working in Scala (2.11.8) and using the Flink framework (1.8.3). I am trying to read a csv file that contains my training data…
A. T.
  • 11
  • 2
1
vote
2 answers

Apache Flink - How Checkpoint/Savepoint works If we run duplicate jobs (Multi Tenancy)

I have multiple Kafka topics (multi tenancy) and I run the same job run multiple times based on the number of topics with each job consuming messages from one topic. I have configured file system as state backend. Assume there are 3 jobs running.…
Raghavendar
  • 159
  • 1
  • 11
1
vote
1 answer

FLINK ,trigger event based on JSON dynamic input data ( like map object data)

I would like to know if FLINK can support my requirement, I have gone through with lot of articles but not sure if my case can be solved or not Case: I have two input source. a)Event b)ControlSet Event sample data is: event 1- { "id" :100 …
Ashutosh
  • 33
  • 8
1
vote
1 answer

how to execute sql queries at run time from datastream's map or flatMap in flink

I am new to flink and want to understand how to run my use case with FLINK: Application has three input data source a) historical data b) get all the live events from kafka c) get the control event that will have a trigger condition since the…
Ashutosh
  • 33
  • 8
1
vote
1 answer

Partition the whole dataStream in flink at the start of source and maintain the partition till sink

I am consuming trail logs from a Queue (Apache Pulsar). I use 5 keyedPrcoessFunction and finally sink the payload to Postgres Db. I need ordering per customerId for each of the keyedProcessFunction. Right now I achieve this…
Nischal Kumar
  • 492
  • 7
  • 15
1
vote
1 answer

Apache Flink : Add side inputs for DataStream API

In my Java application, I have three DataStreams. For example, for One stream data is consumed from Kafka, for another stream data is consumed from Apache Nifi. For these two streams Object type is different. For example, Stream-1 object type is…
Azhagesan
  • 217
  • 1
  • 2
  • 12
1
vote
0 answers

flink readCSV thrown back with "org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException:Timeout waiting for connection from pool"

We are using Flink 1.9.0 Dataset API to read CSV files from Amazon S3 Bucket. Facing connection pool timeout most of the times. Following are the configurations at Flink level Reading 19708 objects from s3 in a single go, as we need to apply the…
shilpa sasi
  • 11
  • 3
1
vote
1 answer

Flink batch program output accumulator doesn't work

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); ...... JobExecutionResult jobExecutionResult = env.execute(XXXOffline.class.getName()); int records =…
Qoobee
  • 196
  • 1
  • 1
  • 12
1
vote
0 answers

Flink: How to handle Null Values in Flink especially while reading a file like csv

Flink: How to handle Null Values in Flink especially while reading a file like CSV. I come from spark back ground. So there is null handling in spark. like df.na.fill etc. So I was wondering if there is any such facility in Dataset API or the only…
Bishamon Ten
  • 479
  • 1
  • 6
  • 20
1
vote
1 answer

Apache Flink - Dataset api - Side outputs

Does Flink supports Side Outputs feature in Dataset(Batch Api) ? If not, how to handle valid and invalid records when loading from file ?
Saravanan
  • 19
  • 4
1
vote
1 answer

Consume multiple text files with Apache Flink DataSet API

I am writing a batch job with Apache Flink using the DataSet API. I can read a text file using readTextFile() but this function just read one file at once. I would like to be able to consume all the text files in my directory one by one and process…
Salvador Vigo
  • 397
  • 4
  • 16
0
votes
0 answers

Flink job listener - not returning job execution result in openshift

Trying to use the flink job listener for notifying batch end. But the job is running in openshift as default detached mode. How to get the job status and other metrics on job executed in job manager. Tried all possible ways.its not coming to job…
Praveen M
  • 1
0
votes
0 answers

Flink 1.15: Error setting BATCH execution mode for DataStream API

I am using Flink 1.15 DataStream api to do ETL job. I want to set my job set BATCH execution mode, so I use code provided in official webstie. env.setRuntimeMode(RuntimeExecutionMode.BATCH); However, I encountered the following error: …
sophia wu
  • 1
  • 1
1 2
3
…
10 11