FlinkCEP - complex event processing for Apache Flink
Questions tagged [flink-cep]
368 questions
0
votes
1 answer
Apache Flink join different DataStreams on specific key
I have two DataStreams, the first one called DataStream source which receive records from a message broker, and the second one is a SingleOutputOperator events, which is the result of mapping the source into Event.class.
I have a uses…
Alter
- 903
- 1
- 11
- 27
0
votes
2 answers
Is it possible for Flink CEP SQL to run on a hopping window
My typical use case is to keep monitoring an event stream to detect event patterns, but I expect the window is a hopping one. By default, interval ... in CEP SQL should define a tumbling window. Is it possible to have a hopping window for CEP…
Grant
- 500
- 1
- 5
- 18
0
votes
1 answer
initializing Sate cause this error "java.lang.NullPointerException: Keyed state can only be used on a 'keyed stream'"
I am trying to initialize a "ListState" inside CoProcessFunction, however it keeps throwing this error
""java.lang.NullPointerException: Keyed state can only be used on a 'keyed stream'"".
this is the code sinnpet that i used
…
abdo_mah90
- 83
- 1
- 7
0
votes
1 answer
System time based event trigger in flink
How can set a trigger in Flink to perform some operation when a particular time has passed?
Eg: Sum of the stream at 1 PM everyday
Abhinav Sharma
- 27
- 8
0
votes
1 answer
How does Flink handle expired keys with CEP
I have streaming job which listens to events, does operations on them using CEP.
Flow is
stream = source
.assignTimestampsAndWatermarks(...)
.filter(...);
CEP
.pattern(stream.keysBy(e-> e.getId()), pattern)
…
ardhani
- 303
- 1
- 11
0
votes
0 answers
Why is apache flink checkpoint size very large?
I've simple Apache Flink job:
**DataSource (Apache Kafka) - Filter - KeyBy - CEP Pattern (with timer) - PatternProcessFucntion - KeyedProcessFunction (*here I've ValueState(Boolean) and registering timer on 5 minutes. If a valueState not null I'll…
0
votes
1 answer
Is there a way to discard partial matches?
I'm monitoring the movement of vehicles. Whenever they enter or exit a geofence, an event is triggered. Via Flink I want to check whether a vehicle was within a fence for at least 2 hours.
Here's the pattern I'm…
tboltson
- 3
- 1
0
votes
1 answer
Flink CepOperator
I am reading the source code of CepOperator from Flink CEP and have a question about the following code snippet:
public void processElement(StreamRecord element) throws Exception {
if (isProcessingTime) {
... ...
}…
Grant
- 500
- 1
- 5
- 18
0
votes
1 answer
Flink task managers are not processing data after restart
I am new to flink and i deployed my flink application which basically perform simple pattern matching. It is deployed in Kubernetes cluster with 1 JM and 6 TM. I am sending messages of size 4.4k and 200k messages every 10 min to eventhub topic and…
user13906258
- 161
- 1
- 13
0
votes
1 answer
Share state between operators (KeyedProcessFunction and RichMapFunction) when RocksDB
Based on the image above, I need to share the state between two operators, at the moment, from one KeyedProcessFunction that will manage to process events and convert them from class X to class Y and keep the states for the incoming records to send…
Alter
- 903
- 1
- 11
- 27
0
votes
1 answer
Apache Flink - FsStateBackend - How state is recovered in case of Task Manager failure which stores state in its local file system
Assume we have 2 Job Managers (ZooKeeper for HA) and 3 Task Managers. I have configured FsStateBackend for checkpointing. I assume that the FsStateBackend runs in each of the Task Managers which maintains the state in the memory. Upon checkpoint,…
Raghavendar
- 159
- 1
- 11
0
votes
1 answer
Flink Checkpointing mode ExactlyOnce is not working as expected
I am newbie to flink apologize if my understanding is wrong i am building a dataflow application and the flow contains multiple data streams which check if the required fields are present in the incoming DataStream or not. My application validate…
user13906258
- 161
- 1
- 13
0
votes
1 answer
Failed to restore checkpointed states using Flink Processor API
The main program is consuming kafka events, then filter -> map -> keyBy -> CEP -> sink. I wrote another separate simple program to read checkpoint directory like the following:
object StateReader extends App {
val path = "file://...."
val env…
Grant
- 500
- 1
- 5
- 18
0
votes
1 answer
Illegal access to create StreamTableEnvironment with JDK 9 in debian
I'm creating a StreamTableEnvironment in my Flink 1.11 application, and this works in Windows, but then, when I'm trying to deploy mi jar in Debian with JDK9(same version in Windows) it throws this error:
Exception in thread "main"…
Alter
- 903
- 1
- 11
- 27
0
votes
1 answer
Watermark fell far behind in Flink CEP
I am using Flink CEP to detect patterns against events from Kafka. For simplicity, events only have one type. I am trying to detect the change in the value of a field in the continuous event stream. The code looks like the following
val streamEnv =…
Grant
- 500
- 1
- 5
- 18