1

I'm working on a project which needs to process a large number of events per second. The project uses Drools 6.5 running in stream mode. The data is fed to the engine as "event" objects.

Due to the large number of events that need to be processed, automatic memory management provided by Drools simplifies the development process significantly. However, drools has a somewhat vague documentation in this category. I need to count the number of events with certain conditions in the past T seconds and fire a rule if the number surpasses a threshold. I currently use sliding windows to achieve this. The problem is, Drools discards events before T seconds passes from their insertion (using @expires value) or does not discard them at all (if @expires tag is removed); thus either making inference impossible, or causing a heap memory overflow in the long run.

Is there a better approach to the problem? Can anyone clarify how inferred expiration works? Am I doing something wrong? Any help would be greatly appreciated.

fzwd
  • 61
  • 10
  • Why don't you use an @expires value that is greater than T seconds? If, however, you are doing this and events disappear earlier there is a bug you should report. – laune May 07 '17 at 12:41
  • I don't want to do that because the application is quite memory-hungry, and the rule set is dynamic and diverse so finding the optimal T is not an easy task – fzwd May 07 '17 at 14:41
  • Then your question is quite misleading. Setting an @expires to less that T and wondering that events cannot be evaluated in a T-window is like shooting the cock in the evening and wondering why he doesn't crow in the morning. – laune May 08 '17 at 05:30
  • This question is a bit too vague to provide a concrete help, but anyway you may find helpful the [Soft expiration for events](https://docs.jboss.org/drools/release/7.0.0.CR3/drools-docs/html_single/index.html#_soft_expiration_for_events) introduced on 7.x – tarilabs May 08 '17 at 06:57
  • I do not use @expires tag, laune. I use sliding time window to count events with specific parameter values that happened in the past T seconds, and I wanted the engine to remove those if they can't participate in the activation of any rule. – fzwd May 13 '17 at 09:12
  • Thanks tarilabs, I will look into that. But unfortunately I'm using Drools 6.5 – fzwd May 13 '17 at 09:13

1 Answers1

1

After a few hours of exploring the documentation for Drools 6.5, I finally found out what was happening. I will leave the information here to help anyone else who might have the same problem.

Important

An explicit expiration policy for a given event type overrides any inferred expiration offset for that same type.

As the documentation says(9.8.1), explicit @expires tag overrides any inferred expiration offset, so in order to let the engine handle the events' life cycle, do not use this tag.

7.5.1. Passive Mode

With Passive mode not only is the user responsible for working memory operations, such as insert(), but also for when the rules are to evaluate the data and fire the resulting rule instantiations - using fireAllRules()

Apparently, in order to use the inferred expiration feature, one can not use the passive execution mode. Running kSession.fireUntilHalt() runs the engine in active mode and enables the use of inferred expiration offsets.

TL;DR: 1. Remove any @expires tag 2. run the engine using fireUntilHalt() in a dedicated thread.

fzwd
  • 61
  • 10
  • You may learn that this may not solve your memory problem. Determining the expiry time automatically for a certain event (fact) type depends on the sum of all rules permitting the derivation of this limit which isn't a cut and dried matter. – laune May 08 '17 at 05:27
  • No, doc7.5.1 does NOT imply your 2nd tldr, and does NOT imply "_in order to use the inferred expiration feature, one can not use the passive execution_". The inferred expiration works also when using Passive Mode, and if you found a case where this is not working as expected, kindly submit bug and reproducer at https://issues.jboss.org/projects/DROOLS , please. – tarilabs May 08 '17 at 06:55
  • I have tested this with some experimentation. The passive mode's inferred expiration did not work while using the active mode fixed the problem. – fzwd May 13 '17 at 09:15
  • @laune I tested the expiration of events with a set of very basic rules. Are you suggesting that automatic expiration might stop working as the rule set gets bigger? – fzwd May 13 '17 at 09:17
  • The number of rules isn't essential, merely the way they rely on the relations between the timestamps of events. It's easily possible to write a rule that forbids automatic expiration. – laune May 13 '17 at 11:59