0

Does Flink guarantee the order of execution of a stream?

I have two Kafka topics (each with single partition), stream 1 and stream 2, and use keyBy, the streams are processed by a coprocess function.

During my testing I can see the the contents of both streams are not always executed in order.

I can set parallelism to 1 to get around this issue, but wanted to understand if what I am seeing is what is expected?

source1.keyBy(e -> e.f0);
source2.keyBy(e -> {
    return new JSONObject(e.f1).getString("someOtherKey");
});

source1.connect(source2)
       .process(new KeyedCoProcessFunction())
       .uid("uid")
       .name("name");
Peter Csala
  • 17,736
  • 16
  • 35
  • 75
gibo
  • 527
  • 2
  • 5
  • 22

1 Answers1

2

Events will maintain their relative ordering if they take the same path through the execution graph. Otherwise, all bets are off.

Here's an example of what can happen with a keyBy. Initally the events are ordered as 1, 2, A

enter image description here

but a keyBy sends A down a different path than 1 and 2:

enter image description here

after which we might see A, 1, 2 (as shown here), or 1, A, 2, or 1, 2, A.

enter image description here

David Anderson
  • 39,434
  • 4
  • 33
  • 60