Stack Overflow
Stack Overflow

Questions tagged [flink-cep]

FlinkCEP - complex event processing for Apache Flink

368 questions
4
votes
3 answers

How to configure Flink to use S3 for backend state and checkpoints

I have a setup with Flink v1.2, 3 JobManagers, 2 TaskManagers. I want to use an S3 bucket instead of hdfs for backend state and checkpoints and zookeeper storageDir fs.s3.accessKey: [accessKey] fs.s3.secretKey: [secretKey] state.backend:…
razvan
  • 559
  • 7
  • 23
4
votes
2 answers

Flink Table API not able to convert DataSet to DataStream

I am using Flink Table API using Java where I want to convert DataSet to DataStream .... Following is my code : TableEnvironment tableEnvironment=new TableEnvironment(); Table tab1=table.where("related_value <…
Akki
  • 493
  • 1
  • 11
  • 23
3
votes
2 answers

How to run multiple complex rules with flink

Here's our use case : We are planning to build a rule based engine on top of flink with huge number of rules(1000s). the rules could be stateless or stateful. Example stateless rule is : A.id = 3 && A.name = 'abc' || A.color = red. Example stateful…
voidMainReturn
  • 3,339
  • 6
  • 38
  • 66
3
votes
1 answer

Apache Flink: How to sink events to different Kafka topics depending on the event type?

I was wondering if it was possible to use the Flink Kafka sink to write events different topic depending on the type of events? Let's say that we have different type of events: notification, messages and friend requests. We want to stream these…
TheEliteOne
  • 105
  • 1
  • 11
3
votes
1 answer

how to stream a json using flink?

i 'm actually working on a stream, receiving a bunch of strings and need to make a count of all the strings. the sums is aggragated, that mean for the second record the sum was added to the day before the output must be some json file looking…
TheEliteOne
  • 105
  • 1
  • 11
3
votes
1 answer

Adding patterns dynamically in Apache Flink without restarting job

My use case is that I want to apply different CEP patterns to the same datastream. the CEP patterns come dynamically & i want them to be added to flink without having to restart the job. While all conditions can be handled via custom classes that…
Rohit Gupta
  • 31
  • 3
3
votes
1 answer

Flink CEP state store

How does Flink CEP manage the intermittent states ? where does it store them ? Is it just in-memory or is there a fast persistent storage backing the states ? The documentation does not mention this anywhere.
Mahesh
  • 133
  • 6
3
votes
2 answers

The generic type parameters of 'Map' are missing in Flink-CEP

Code for detecting pattern in Flink-CEP is shown below // Generate temperature warnings for each matched warning pattern DataStream warnings = tempPatternStream.select( (Map pattern) -> { …
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
3
votes
1 answer

Flink CEP: Which method to join data streams for different type of events?

Suppose that I have 2 different types of data streams, one providing weather data and the other providing vehicle data, and I would like to use Flink to do complex event processing on the data. Which method in Flink 1.3.x is the correct method to…
Jamie
  • 91
  • 1
  • 2
  • 6
3
votes
1 answer

When might flink support pattern match involving fields of previous events?

It would be great to be able to match events based on their field values going beyond the current capability of creating patterns out of events matching separate criteria. For example as explained at…
Systems User
  • 31
  • 1
3
votes
1 answer

is it possible to add new patterns in FLINK CEP after calling execute()?

My code is the following: StreamExecutionEnvironment env= StreamExecutionEnvironment.getExecutionEnvironment(); DataStream input = env.addSource(new MyCustomSource()); Pattern pattern =…
Ghdiri Abdallah
  • 33
  • 3
3
votes
1 answer

How can I add/change new patterns/queries dynamically/at runtime in apache Flink?

I am trying to build a CEP system with Apache Flink for correlating events. One of the requirements is to be able to add new patterns for anomaly detection at runtime without losing the system availability. Any Ideas of how could I do that? For…
Pedro Chaves
  • 123
  • 13
3
votes
1 answer

Flink Window and State Maintenance

I'm working on apache flink for data streaming and I have few questions. Any help is greatly appreciated. Thanks. 1) Are there any restrictions on creating tumbling windows. For example, if I want to create a tumbling window per user id for 2 secs…
Neoster
  • 195
  • 2
  • 11
3
votes
2 answers

How to specify different log4j.properties file for different flink job running on same standalone cluster

I have multiple flink job running on a standalone cluster. I want different log files for different flink job. So how can I pass different log4j.properties file while submit flink job.
Anand Shaw
  • 229
  • 5
  • 13
3
votes
0 answers

flink streaming - cep Could not find previous shared buffer entry with key

Im trying to run cep on Flink,and the test data from local path, at the beganing,I was set the file's size about 1G, it run well.But when I was set the file's size to 10G,it was occured a question below. Exception in thread "main"…
Jary zhen
  • 437
  • 6
  • 18
1
2
3
…
24 25