Stack Overflow
Stack Overflow

Questions tagged [flink-cep]

FlinkCEP - complex event processing for Apache Flink

368 questions
2
votes
2 answers

Flink dynamic scaling 1.5

This is regarding dynamic rescaling in Flink 1.5 I am using Yarn for running Flink jobs. I start these jobs with a static resource. Is there any option to scale out these job by itself in specific conditions like if there's a memory issues. In…
user3107673
  • 423
  • 4
  • 9
2
votes
1 answer

Why CEP doesn't print the first event only after I input second event when using ProcessingTime?

I sent one event with isStart true to kafka ,and made Flink consumed the event from the kafka, also set the TimeCharacteristic to ProcessingTime and set within(Time.seconds(5)), so I expected that CEP would print the event after 5 seconds I sent the…
YuFeng Shen
  • 1,475
  • 1
  • 17
  • 41
2
votes
2 answers

Apache Flink CEP how to detect if event did not occur within x seconds?

For example A should be followed by B within 10 seconds. I know how to track if this DID occur (.next, .within), but I want to send an alert if B never happened within the window. public static void main(String[] args) throws Exception { …
atkayla
  • 8,143
  • 17
  • 72
  • 132
2
votes
0 answers

How to Implement the Flink CEP using time window?

I want to implement Event Alert using Flink CEP. My use case: I want to apply time Window for one hour and within that time if device usage is greater than some threshold(100) then I want to alert the end user and fire that alert only once in…
NIrav Modi
  • 6,038
  • 8
  • 32
  • 47
2
votes
1 answer

Is there a foldLeft operator equivalent on a 1.4 Apache Flink DataStream

I have a stream setup like this using Apache Flink 1.4 starts with DataStream then keyBy then window then aggregate the output of the aggregate operation is an AggregateResult object so the stream as it ends now is…
jwe4
  • 41
  • 2
2
votes
1 answer

Joining more than 2 streams in Flink & applying CEP to more than 2 streams without joining them

Question # 1 : I am working on a case scenario where we need to fuse data from multiple sensors [8 sensors for instance] and join them in a tree form. For example, joining [s1,s2,s3 s4] to form stream A and then [s5,s6,s7 and s8] to form Stream B…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
2
votes
3 answers

Naming operators, sources, sinks and patterns to be shown in Flink Execution Plan UI

I just want to give names to operators and sources. Let's take an example This is a screenshot of the execution plan, that I have taken form Flink Dashboard. Here I have 2 sources of DataStreams and then I am joining them. My question is that, can…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
2
votes
1 answer

Unable to run Flink job when jar file is submitted, but program running otherwise in Intellij

My Flink Program runs successfully in IntellijIdea, but when I make a jar file of that program to submit it as jar, it shows me the following error ava.lang.RuntimeException: Could not look up the main(String[]) method from the class…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
2
votes
2 answers

getting numOfRecordsIn using counters in Flink

I want to show numRecordsIn for an operator in Flink and for doing this I have been following ppt by data artisans at here. code for the counter is given below public static class mapper extends RichMapFunction{ public …
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
2
votes
2 answers

Is it possible to process multiple streams in apache flink CEP?

My Question is that, if we have two raw event streams i.e Smoke and Temperature and we want to find out if complex event i.e Fire has happened by applying operators to raw streams, can we do this in Flink? I am asking this question because all the…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
2
votes
1 answer

Flink Streaming AWS S3 read multiple files in parallel

I am new to Flink, my understanding is following API call StreamExecutionEnvironment.getExecutionEnvironment().readFile(format, path) will read the files in parallel for given S3 bucket path. We are storing log files in S3. The requirement is to…
Abirami
  • 87
  • 7
2
votes
1 answer

Flink CEP: How to check if next event didn't appear in given time window?

I'm trying to find out how to implement the following logic having stream of events: First event appear in given time window(30s) Second event that is related to the first one(by some property) didn't appear in given time window Generate new error…
jurek
  • 63
  • 7
2
votes
1 answer

Flink CEP No Results Printed

I am trying to print out a string if Hello and world are found using the Flink CEP library. My source is Kafka and using the console-producer to input the data. That part is working. I can print out what I enter into the topic. However, it will…
Crackerman
  • 717
  • 1
  • 9
  • 22
2
votes
2 answers

Flink CEP is not deterministic

I have the following code running locally without a cluster: val count = new AtomicInteger() val env = StreamExecutionEnvironment.getExecutionEnvironment env.setParallelism(1) val text: DataStream[String] =…
Sriram
  • 146
  • 7
2
votes
1 answer

Create new datastream dynamically in Flink

I am trying to create new datastreams out of one raw datastream based on certain rules available in database. Once I start the Flink job and after some, if I need to create a new datastream as per updated entries in database, how can I achieve that.…
Anand Shaw
  • 229
  • 5
  • 13
1 2 3
…
24 25