How to delete old keys from a KTable

Question

I have a producer which publishes records with a key of either A or B to a Kafka topic.

In a streams app, I'm flat-mapping each record with key A to records with key U, V or W, and each record with key B to records with key X, Y or Z.

The number of records created by each flat map operation varies. E.g., a particular record with key A might be mapped to one record with key U, but another might be mapped to one of key V and one of key W.

I want to create a KTable for these flat-mapped records. However, I want the keys of this KTable to match the keys generated by the last flat-map operations for each record of key A and B.

E.g., if, at a particular moment, the KTable had keys of X, Y, and U and then a record with key B was published and mapped to just one record with key X, I would want the KTable to just have keys for X and U.

If anyone has any suggestions for how I can do this, I'd be very grateful.

Answer 1

Your flatMap() will need to be stateful, ie, you can use flatTransfrom() with state store to implement it. The result of flatTransform() can we upserted into the result KTable via KStream#toTable() operator (added in 2.5 release).

The key-value store maintains the raw data from the input topic. Each time the key changes, you now have access to the old record (from the store) and new record (input to tranform()) for the key and can emit corresponding record to update the downstream KTable.