Excessive memory usage in esper while aggregating values over 60 minutes using a context

Question

I want to aggregate sensor values for each hour of the day. The time information comes from timestamps in the event stream. To that end, I've created four EPL statements:

1. The first statement declares a context that partitions events by sensorId
2. The second statement detects when the hour-of-day changes in the event stream of a particular sensor and sends an event when that occurs
3. The third statement is a context declaration that partitions events by sensorId and hourOfDay. The lifetime of the context is controlled by the events published by the second statement.
4. The last statement makes use of the context created in statement #3 and calculates a sum over the sensor values.

I'm receiving events from approximately 300 sensors every 2-6 seconds. I have esper 6.1.0 embedded in a java application with a heap size of 8 GB. After about 15 minutes, the memory pressure is such that garbage collection goes into overdrive, rendering the application unusable. If I remove the last EPL statement, my application behaves normally again.

I'm somewhat perplexed by this behavior. I thought that when a context is used, esper would not accumulate events in memory.

So here's my question: Using EPL, how do I perform a simple aggregation (e.g. sum, average) such that the memory consumption is O(n) rather than O(n*t), where n is the number of sensors and t is the length of the time window?

For concreteness, here are my EPL statements.

Statement 1:

create context ctxPartitionById partition by sensorId from SensorEvent

Statement 2:

context ctxPartitionById
INSERT INTO HourChanged
SELECT
  E.sensorId as sensorId,
  prev(1, E.occurredAtHour) AS lastHour,
  E.occurredAtHour AS currentHour
FROM SensorEvent#length(2) E
WHERE E.occurredAtHour != prev(1, E.occurredAtHour)

Statement 3:

create context ctxPartitionByIdAndHour
  context PartitionedByKeys
    partition by
      sensorId, currentHour from HourChanged,
      sensorId, occurredAtHour from SensorEvent,
  context InitiateAndTerm
    initiated by HourChanged as e1
    terminated by HourChanged(sensorId=e1.sensorId and lastHour=e1.currentHour)

Statement 4:

context ctxPartitionByIdAndHour
SELECT
  E.sensorId,
  E.occurredAtHour,
  SUM(E.value) AS sensorValueSum
FROM SensorEvent E
output last when terminated

sensorId is an instance of a java class that implements Comparable.

occurredAtHour is a java.util.Date instance. Its value is a timestamp that has been rounded down to the hour.

value is a double.

Answer 1

The keyed context does not forget partitions. So it you do "partition by sensorId, currentHour from HourChanged, sensorId, occurredAtHour from SensorEvent" and since this is a keyed context (at the top) partition keys stay around since the engine thinks you are always looking for the next HourChanged event for that same key. This means that every possible combination of "sensorId" and "currentHour" are being remembered by the engine. So this may not be what you want. Tips for this are in the solution patterns page.

For statement 4, output rate limiting can causes events to stay in memory since by default the engine computes the output when terminated. Note how your select clause selects event properties. When you select event properties these come from actual event(s). Instead you could simply make this a fully-aggregated query and so it won't have a reason to keep ANY event.

context ctxPartitionByIdAndHour
SELECT last(E.sensorId), last(E.occurredAtHour), SUM(E.value) AS sensorValueSum
FROM SensorEvent E
output last when terminated

You could do something like in the solution pattern. http://espertech.com/esper/solution_patterns.php#absence-11 http://espertech.com/esper/solution_patterns.php#semantic-window-2

Answer 2

Adding @Hint('enable_outputlimit_opt') to the fourth and last EPL statement solves the issue.

Apparently esper retains events in memory when output last is used, unless this particular hint is present. See 5.7.3. Runtime Considerations in the esper docs.