0

I have a use case in which I don't want to drop the record from the KStream if KTable key doesn't exist & I want to wait for the KTable key (record) to arrive & then sending fully populated data to a new kafka topic.

I have found a reference in CustomStreamTableJoinIntegrationTest.java https://github.com/confluentinc/kafka-streams-examples/blob/5.2.1-post/src/test/java/io/confluent/examples/streams/CustomStreamTableJoinIntegrationTest.java

But seems like this is not a very good approach to hold the record manually in a custom streamBufferStore & after the record comes to KTable then finally sending fully populated data to a new kafka topic & manually deleting the data from the buffer store.

Is there any better way to handle this use case.

final List<KeyValueWithTimestamp<String, Double>> inputStreamRecords = Arrays.asList(
    new KeyValueWithTimestamp<>("alice", 333.33, TimeUnit.MILLISECONDS.toMillis(5)),
    new KeyValueWithTimestamp<>("alice", 999.99, TimeUnit.MILLISECONDS.toMillis(10)),
    new KeyValueWithTimestamp<>("bobby", 222.22, TimeUnit.MILLISECONDS.toMillis(15)),
    new KeyValueWithTimestamp<>("alice", 555.55, TimeUnit.MILLISECONDS.toMillis(30)),
    new KeyValueWithTimestamp<>("alice", 666.66, TimeUnit.MILLISECONDS.toMillis(40))
);

final List<KeyValueWithTimestamp<String, Long>> inputTableRecords = Arrays.asList(
    new KeyValueWithTimestamp<>("alice", 1L, TimeUnit.MILLISECONDS.toMillis(20)),
    new KeyValueWithTimestamp<>("alice", 2L, TimeUnit.MILLISECONDS.toMillis(39)),
    new KeyValueWithTimestamp<>("bobby", 8L, TimeUnit.MILLISECONDS.toMillis(400))
);

final List<KeyValue<String, Pair<Double, Long>>> expectedOutputRecords = Arrays.asList(
    new KeyValue<>("alice", new Pair<>(333.33, 1L)),
    new KeyValue<>("alice", new Pair<>(999.99, 1L)),
    new KeyValue<>("alice", new Pair<>(555.55, 1L)),
    new KeyValue<>("alice", new Pair<>(666.66, 2L)),
    new KeyValue<>("bobby", new Pair<>(222.22, null))
);
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
Sharry India
  • 341
  • 1
  • 9

0 Answers0