FlinkCEP - complex event processing for Apache Flink
Questions tagged [flink-cep]
368 questions
0
votes
2 answers
How to recover a KeyedStream from different filters applied after been keyed before
How can I spread out the same keyedStream and apply filters according to different uses cases without the need to create a new keyedStream at the end of the filtering?
Example:
DataStream streamFiltered =…
Alter
- 903
- 1
- 11
- 27
0
votes
1 answer
Checkpoints increasing over time in Flink
in aggregation to this question I'm still not having clear why the checkpoints of my Flink job grows and grows over time and at the moment, for about 7 days running, these checkpoints never gets the plateau.
I'm using Flink 1.10 version at the…
Alter
- 903
- 1
- 11
- 27
0
votes
1 answer
How to understand the storage of data in cep？
Pattern pattern = Pattern.begin("start")
.next("middle").where(new SimpleCondition() {
@Override
public boolean filter(Event value) throws Exception {
return value.getName().equals("error");
…
MARS
- 1
- 1
0
votes
1 answer
Flink CEP: FollowedBy pattern: same event loops the pattern multiple times
I am new to Flink CEP and have been playing around with the patterns for better understanding of them.
I have a simple case of a "begin" & a followedBy".
I notice that in the case of followedBy, the same event is looping through it multiple…
Rishi Sarkar
- 11
- 3
0
votes
1 answer
About Flink 1.11 Deployment and Setup
I need to run a Flink job in production, I have just one machine (is not a cluster). I'm trying to setup my Flink job for production and do it in the best way possible, different as I have it right now (running as a java application java -jar…
Alter
- 903
- 1
- 11
- 27
0
votes
2 answers
FLINK - will SQL window flush the element on regular interval for processing
I am confused if TUMBLE window will get calculated on regular interval and emit the elements for processing. example I have a query that is expected to work on interval 10 second.
select id, key from eventTable GROUP BY TUMBLE(rowTime, INTERVAL…
Ashutosh
- 33
- 8
0
votes
1 answer
Flink ProcessFunction onTimer first event LONG.MIN_VALUE
I am having a use case: stepA -> stepB -> stepC ... Now I need to use Flink processfunction to monitor the track. for example, once stepA comes in, I set up a timer 10s after A, and when 10s has passed, that timer is triggered, and we check our…
0
votes
1 answer
Metrics exporter
Is there a way to export metrics to Prometheus about state or checkpoint sizes per operator? I know that I can find this values from the Flink Dashboard, but I need to have all the metrics in just one Grafana board with Prometheus, hoping this is…
Alter
- 903
- 1
- 11
- 27
0
votes
1 answer
FLINK- how to process logic on sql query result
My requirement is to process or build some logic around the result of sql query in flink. For simplicity lets say I have two sql query they are running on different window size and one event stream. My question is
a) how I will know for which query…
Ashutosh
- 33
- 8
0
votes
1 answer
Apache-Flink 1.11 Unable to use Python UDF through SQL Function DDL in Java Flink Streamming Job
In Flip-106 there is an example of how to call a user-defined python function in a batch job java application through SQL Function DDL...
BatchTableEnvironment tEnv =…
Jonathan Figueroa
- 31
- 2
0
votes
1 answer
Causes of CPU load on the machine increase when flink run more more than 12 hours
I have a flink job, with parallelism set to 6, few simple transformations and the issue is that when Flink is been running for more than 12 hours for example the Load on the machine start to increase, then I thought that was because of the traffic…
Alter
- 903
- 1
- 11
- 27
0
votes
1 answer
How does TM recovery handle past broadcasted data
In the context of HA of TaskManagers(TM), when a TM goes down a new one will be restored from latest checkpoint of faulted by the JobManager(JM).
Say we have 3 TMs (tm1, tm2, & tm3)
At a give time t where everyone's checkpoint(cp) is at cp1. All TMs…
ardhani
- 303
- 1
- 11
0
votes
2 answers
Which set checkpointing interval (ms)?
everyone.
Please help me.
I write apache flink streraming job, which reads json messages from apache kafka (500-1000 messages in seconds), deserialize them in POJO and performs some operations (filter-keyby-process-sink). I used RocksDB state…
0
votes
1 answer
Flink combination of windowByTime and triggerByCount
source.keyBy(0)
.window(TumblingEventTimeWindows.of(Time.seconds(5)))
.trigger(PurgingTrigger.of(CountTrigger.of[TimeWindow](2)))
.process(new TestFun())
Explanation:
Let's say I have 3 events[E1, E2, E3], which should be trigger by…
Tulasi
- 79
- 1
- 9
0
votes
1 answer
How in Flink CEP can we detect a pattern that last a period of time?
I want to detect a pattern with Flink CEP, here my use case:
I should raise an event when i got the speed of my vehicle above a speedLimit for a laps of time.
Example1: (speedlimit = 100, period=60 seconds)
event1: speed = 50, eventtime=0
event1:…
Khairy
- 81
- 1
- 10