How to make sure tuples with the same key are processed in order

Question

I have made job with Hazelcast Jet that converts a stream of IoT measurements into a stream of alarms.

So, whenever the humidity level of one sensor goes above a threshold, an alarm is raised. When it falls again below the threshold, the alarm is cleared. There can be up to 3 levels of thresholds (severity).

Currently, I have issues when the job is started. It will flush all buffered events from my RabbitMQ source. So, far events are ordered because local parallelism is one (let's assume a single member cluster here). But we the events are dispatched to the pool of cooperative threads, there is no garantee on the order. Can I instruct Jet to process all events with the same sensor ID in order?

Here is the current definition of my pipeline:

  StreamStage<Notification> ss = l
        .drawFrom(
              Sources.<SimpleEntry<String, String>> streamFromProcessor("rabbitmq", ReadRabbitMQP.readRabbitMQ()))
        .map(e -> makeMeasurement(e))
        .flatMap(e -> checkThresholds(e))
        .flatMap(e -> checkNotification(e));

  ss.drainTo(Sinks.logger());

checkNotification compares the severity of the event with the latest severity for this sensor. That's why order is important.

I tried to implement to solution suggested by Gokhan Oner: I modifed the source to ouput SimpleMeasurement objects. This way I can add the timestamp just after the source.

StreamStage<Notification> ss = l
      .drawFrom(Sources.<SimpleEntry<Integer, SimpleMeasurement>> streamFromProcessor("rabbitmq",
                  ReadRabbitMQP.readRabbitMQ(mGroupNames, mLocalParallelism)))
      .addTimestamps(e -> e.getValue().getTimestamp().toEpochMilli(), 1000)
      .flatMap(e -> checkThresholds(e))
      .groupingKey(e -> e.getSensorId())
      .window(WindowDefinition.tumbling(1))
      .aggregate(AggregateOperations.sorting(DistributedComparator.comparing(e -> e.getPeakTime())))
      .flatMap(e -> checkNotification(e));

ss.drainTo(Sinks.logger());

With this code, events are still not processed in order for the same sensor ID. Moreover, there is a 20 seconds delay from the moment the event is read from the source till it is processed in 'checkNotification'.

Do you mind sharing your rabbitmq processor source? I'm curious if you have investigated the new [SourceBuilder](https://docs.hazelcast.org/docs/jet/0.7.2/javadoc/com/hazelcast/jet/pipeline/SourceBuilder.html) API in 0.7. — Can Gencer, Jan 09 '19 at 07:28
@CanGencer The rabbitmq source is here (https://gist.github.com/PeeWee2201/41f634125384f8c1ae3ed99df101c933). No I haven't use the new SourceBuilder API. Would it help in solving my problem? — PeeWee2201, Jan 11 '19 at 16:05

score 1 · Answer 1 · answered Jan 08 '19 at 19:30

1

@PeeWee2201, since this is a distributed stream, there is no guaranteed order. But if you want to process notifications from the same sensors in order, then you need to:

add a timestamp to the events
group by Sensor ID
define a window, 10 sec, 30 sec etc, so that events can be aggregated within this window
Sort all events based on any property you want in the same window

So the job should look like this:

  StreamStage<Notification> ss = l
        .drawFrom(
              Sources.<SimpleEntry<String, String>> streamFromProcessor("rabbitmq", ReadRabbitMQP.readRabbitMQ()))
        .addTimestamps(...., ...)
        .groupingKey(....)
        .window(WindowDefinition.tumbling(....))
        .aggregate(AggregateOperations.sorting(....))

If makeMeasurement(e) is a step that converts the data & can run in parallel, you can add it before grouping.

After this, you'll have a list of objects for checkThresholds method: All messages within the window for the same sensorId ordered by arrival time or whatever sort order you used.

I believe this'll help to resolve your problem.

answered Jan 08 '19 at 19:30

Gokhan Oner

3,237
19
25

Well, I could not make it work. I had to use a big value for the 'lag' parameter otherwise some events are dropped and the stream is 'corrupted'. This seems strange to me as they always arrive in order from the source. Maybe this is also because Jet does not look at them in order... But a big lag iccurs significant latency, which is also not acceptable. – PeeWee2201 Jan 11 '19 at 16:10
@PeeWee2201, is there a timestamp in the messages that you're adding since the message is just a String value & how big of a leg you need to add, you think? – Gokhan Oner Jan 11 '19 at 19:54
I tried to implement your solution. I have updted the question with the results. Unfortunately it does not work. – PeeWee2201 Jan 12 '19 at 10:45
Can you share some log output? Your window is 1 millisecond so there is no way that it would delay the processing for 20 seconds. Also, of you can share a sample message & SampleMeasurement objects, I want to try the same using your RabbitMQ source – Gokhan Oner Jan 12 '19 at 18:20
I put up a sample project that exhibits this behavior. You can find it here: https://bitbucket.org/pduflot/hazelcast-jet-demo.git. First time it is started, nothing will happen. The next times it will process 100 events enqueued at startup. You can see in the log output that order is not enforced. – PeeWee2201 Jan 14 '19 at 11:12
@PeeWee2201, some observations: You're producing notifications 10 seconds apart, your window size is 10 millis & your lag is 1 sec. These values don't make much sense. Generate data points for each 10 milliseconds, change lag to 10 milliseconds & change your window size > 10 milliseconds, 1 tried both 100 and 1000, and you'll see that it's working as expected. – Gokhan Oner Jan 14 '19 at 13:05
I cannot change the period for the events. It is defined by the sensors. I can change the size of the window to 100 seconds. In this case, yes, it is correct BUT the last window is not closed. There is a one minute delay before the last events go through the pipeline. This means that under normal conditions, I will have 100 seconds latency between events produced and notification triggered :-( Any ideas? – PeeWee2201 Jan 14 '19 at 15:13
You can reduce the window size to 20 sec for example for again, you’ll get results delayed. With windowing aggregation, there is nothing you can do. You can ser the window size to 10 second as well but then you’ll just get 1 event for same device so it doesnt make sence. I suggest you open a github issue to Hazelcast-Jet so the team may suggest something else or may add support for this in later releases if possible. – Gokhan Oner Jan 14 '19 at 15:27

How to make sure tuples with the same key are processed in order

1 Answers1