3

Here's our use case : We are planning to build a rule based engine on top of flink with huge number of rules(1000s). the rules could be stateless or stateful. Example stateless rule is : A.id = 3 && A.name = 'abc' || A.color = red. Example stateful rule is : A is event.id =3, B is event.name = 'abc', C is event.color = red and we are looking for pattern AB*C over time window of 1 hour.

Now we have tried to use flink CEP for this purpose and program crashed because of lot of threads. The explanation is : every application of CEP.pattern creates a new operator in the graph and flink can't support that many vertices in a graph.

Other approach could be to use processFunction in flink, but still to run the rules on events stream, you'd have to use some sort of CEP or write your own.

My question is, does anybody have any other suggestions on how to achieve this ? Any other CEPs that integrate and work better with flink (siddhi, jasper, drools) ? Or are we thinking this completely wrong and the only way to achieve this is to keyBy ruleID, so that 1 rule runs per task slot(but in this approach all data gets sent to all slots) ? Any experience/suggestion would be helpful.

voidMainReturn
  • 3,339
  • 6
  • 38
  • 66

2 Answers2

1

Unlike regular programming libraries, all big data processing engines creates new pipelines and operators for queries and programs (like Spark & Flink).

We had a similar problem in our rule engine, we have to commonize similar rules with parameter based ones. In that style we had a few pipelines that apply rule restrictions on the rules. We were using a home grown software for that I can not mention here. I think you can also take a similar approach here.

veysiertekin
  • 1,731
  • 2
  • 15
  • 40
1

You might have a look at siddhi CEP. It has a flink connector but I'm not sure how well it is maintained.

This year there was a talk on flink forward too!

Otherwise you might need to implement it by your self with a broadcast state pattern.

emilio
  • 588
  • 3
  • 12