is it possible to add new patterns in FLINK CEP after calling execute()?

Question

My code is the following:

StreamExecutionEnvironment env= StreamExecutionEnvironment.getExecutionEnvironment();

DataStream<MyObject> input = env.addSource(new MyCustomSource());

Pattern<MyObject, ?> pattern = Pattern.<MyObject>begin("start");

PatternStream<MyObject> patternStream = CEP.pattern(input, pattern);

... define my pattern

DataStream<MyObject> resultStream = patternStream.select(new MyCustomPatternSelectFunction());

resultStream.addSink(new MyCustomSinkFunction(subscriptionCriteria));

try
  {
    env.execute();
  }
  catch (Exception exception)
  {
    log.debug("Error while ", exception);
  }

this code works and does what i want and i get a result stream that follows the pattern i set.

what i want to know is if it is possible to apply new patterns to this source i added to the environment later when i wish and thus getting different result stream matching different patterns without calling env.execute() another time because when i do that in addition to my new result stream i get redundant old result streams (i.e. old patterns get executed multiple times)?

If I understand you correctly, you want to modify your Flink job after you started its execution? Flink's Savepoints might help you in this use case: http://data-artisans.com/savepoints-part-2-updating-applications/ — twalthr, Dec 05 '16 at 07:52
@twalthr that only works using the command line if a am not mistaken, no? in that case, no that is not what i meant, I wanted to add flink cep patterns dynamically which is apparently not supported natively at the moment and one must implement it himself as per Till's answer. — Ghdiri Abdallah, Dec 06 '16 at 08:53

score 1 · Accepted Answer · answered Dec 05 '16 at 09:36

At the moment Flink's CEP library does not support dynamic pattern changes out of the box. Thus, once you've defined your pattern and started your job, it will only process this defined pattern.

However, you can write your own operator implementing the TwoInputStreamOperator interface which receives on one input pattern definitions and on the other input the stream records (similar to a CoFlatMap function). For every new pattern you would then have to compile a new NFA on the operator and feed any new incoming stream elements to this NFA as well. That way, you could achieve your intended behaviour.

In the future, we will most likely add this feature to Flink's CEP library.

is it possible to add new patterns in FLINK CEP after calling execute()?

1 Answers1