Stack Overflow
Stack Overflow

Questions tagged [flink-batch]

158 questions
0
votes
1 answer

Flink 1.14.3 - [issue] failed to bind to /0.0.0.0:6123

We are using 1.14.3 version of flink and when we try to run Job manager, we are getting below exception. I tried entering akka.remote.netty.tcp.hostname = "127.0.0.1" in flink-conf.yml file and even updated IP with hostname. But didnt…
Vinayraj007
  • 1
0
votes
2 answers

Apache Flink - stateful function for keeping calculation context

I have a use case when I execute some calculations on a part of data and these calculations depend on the context (intermediate state). For example: I have some orders and perform some calculations on them. Calculations are performed for orders…
Ardelia Lortz
  • 72
  • 1
  • 8
0
votes
1 answer

Does Apache Flink cache jobs?

I noticed that each time I run a new job it takes around 20% longer compared to the time when I launch it again? Does flink cache some results and reuses them if a job is run multiple times? If so, how can I control this? I would like to mesure how…
nanobot
  • 108
  • 5
  • 18
0
votes
0 answers

flink batch ends prematurely without a hint why

I am running a simple batch job in Flink. The dashboard says the job is finished but it only shows that about 30000 records were processed out of about 220000. Otherwise, there are no errors and the output seems as expected. How to check why the job…
Ben
  • 1
  • 2
0
votes
1 answer

Apache Flink with Kinesis Analytics : java.lang.IllegalArgumentException: The fraction of memory to allocate should not be 0

Background : I have been trying to setup BATCH + STREAMING in the same flink application which is deployed on kinesis analytics runtime. The STREAMING part works fine, but I'm having trouble adding support for BATCH. Flink : Handling Keyed Streams…
jt97
  • 13
  • 3
0
votes
1 answer

Apache Flink : Batch Mode failing for Datastream API's with exception `IllegalStateException: Checkpointing is not allowed with sorted inputs.`

A continuation to this : Flink : Handling Keyed Streams with data older than application watermark based on the suggestion, I have been trying to add support for Batch in the same Flink application which was using the Datastream API's. The logic is…
jt97
  • 13
  • 3
0
votes
1 answer

Extracting a field value (mostly constant) from Apache Flink Datastream

I have a Datastream which contains some fields like event_id, timestamp, etc. which remains constant for many records in pipeline. I want to use those in filename while writing it back in ParquetFormat using StreamingFileSink. We can use suffix and…
Anish Sarangi
  • 172
  • 1
  • 14
0
votes
1 answer

Apache Flink - Matching with fields having different values in successive patterns

Consider the use case where we need to find the pattern for a attack like 10 failed logons from the same device and same username followed by a success logon from different device but same username. This should happen within 10 mins. Let us say we…
JDForLife
  • 91
  • 2
  • 10
0
votes
1 answer

Flink: Left join equivalent of Dataset API in Batch mode of DataStream API?

It has been mentioned in Flink docs that DataSet API will be deprecated in future. So I am looking into prototyping this Dataset API to DataStream API in Batch Mode (which I believe is in Beta right now) migration. We have this(similar) code in our…
ybd
  • 3
  • 2
0
votes
1 answer

How to stop flink reading duplicate data while reading csv file

I would like to explain my problem statement by explaing below scenario first. Scenario : I am working on continuos file reading using flink's PROCESS_CONTINOUS mode using flink+java8. This is actually a batch reading kind of functionality in which…
MiniSu
  • 566
  • 1
  • 6
  • 22
0
votes
2 answers

Unable to read a (text)file in FileProcessing.PROCESS_CONTINUOS mode

I have a requirement to read a file continously from a specific path. Means flink job should continously poll the specified location and read a file that will arrive at this location at certains intervals . Example: my location on windows machine…
MiniSu
  • 566
  • 1
  • 6
  • 22
0
votes
1 answer

Flink job cant use savepoint in a batch job

Let me start in a generic fashion to see if I somehow missed some concepts: I have a streaming flink job from which I created a savepoint. Simplified version of this job looks like this Pseduo-Code: val flink =…
TobiSH
  • 2,833
  • 3
  • 23
  • 33
0
votes
0 answers

How to keep flink logs for long time?

I am using apache flink 1.11.3 for running my java pipelines. I have noticed that failed job are getting cleaned from the history after few hrs. Can we have any parameter regarding this to retained failed/completed task history for at least 7-10…
Joseph D
  • 189
  • 1
  • 12
0
votes
1 answer

How to keep flink batch job running continuously on local

I am practicing file reading through the flink batch processing mechanism on a Windows 10 machine. I downloaded flink-1.7.2-bin-hadoop24-scala_2.12.tgz from flink's official site and executed start-cluster.bat . I uploaded the jar though Flink's UI…
MiniSu
  • 566
  • 1
  • 6
  • 22
0
votes
1 answer

How to process an already available state based on an event comes in a different stream in flink

We are working on deriving the status of accounts based on the activity on it. We calculate and keep the expiryOn date(which says the tentative, future date on which account expires) based on the user activity on the account. We have a manual date…
Kranthi
  • 37
  • 6
1 2 3
…
10 11