Latest Value from the KStream for a given key

Question

I have KTable like below

KTable<productId1,productValue1>
KTable<productId2,productValue2>
KTable<productId3,productValue2>

Then i have KStream<custId,productId>, the productId is of any of productId1, productId2, productId3. I want to have the corresponding productValue.

The response would be like KStream<custId, productValueN>

If the KStream receives productId as productId2, then the response would be KStream<custId, productValue2>

How this could be possible? Thanks

Sounds like a duplicate of https://stackoverflow.com/questions/59326550/kafka-streams-join-by-key-with-complex-condition/59492835 — Matthias J. Sax, May 25 '20 at 19:10
I want to join the value of KStream with the key with KTable. Something like KeyValueJoin between two KStreams. — Venkata Madhu, May 25 '20 at 22:46
Just use `KStream#selectKey()` (or `map()`) before the join -- you need to set the join attribute as key before the join. — Matthias J. Sax, May 26 '20 at 16:20

ck1 · Answer 1 · 2020-05-31T17:11:19.010

0

You can accomplish this using a GlobalKTable for products, and then joining the customers KStream with it as follows:

Serde<String> stringSerde = new Serdes.StringSerde();

StreamsBuilder streamsBuilder = new StreamsBuilder();

GlobalKTable<String, String> productsTable =
        streamsBuilder.globalTable(PRODUCTS_TOPIC, Consumed.with(stringSerde, stringSerde),
                Materialized.<String, String, KeyValueStore<Bytes, byte[]>>as("products")
                        .withKeySerde(stringSerde)
                        .withValueSerde(stringSerde));

streamsBuilder.stream(CUSTOMERS_TOPIC, Consumed.with(stringSerde, stringSerde))
        .join(productsTable,
                (customerId, productId) -> productId,
                (productName1, productName2) -> productName2,
                Named.as("customersProducts"))
        .print(Printed.toSysOut());

edited May 31 '20 at 17:11

answered May 30 '20 at 19:40

ck1

5,243
1
21
25

Thanks, i was looking for this KeyValueMapper with KTable. Like the way it is available below below for GlobalKTable
** KStream join(GlobalKTable var1, KeyValueMapper super K, ? super V, ? extends GK> var2, ValueJoiner super V, ? super GV, ? extends RV> var3);**
May be selectKey what i should be using,
Anyway Thanks – Venkata Madhu May 31 '20 at 05:27
I've posted another answer for how to do this using a `KTable`. – ck1 May 31 '20 at 17:06

ck1 · Answer 2 · 2020-05-31T17:09:53.620

Unfortunately KTables don't support KeyValueMappers: KStream/KTable joins are performed strictly by comparing keys. GlobalKTables support this, making the whole process of joining much easier (see my other answer).

To get this to work then, you need to rekey the customers topic KStream to use the product ID, so that you can perform the join with the products topic KTable.

Once you've obtained the product name, you can rekey the resulting joined stream back to the customer ID. For example:

// Key: product ID, value: product name
KTable<String, String> productsTable =
    streamsBuilder.table(PRODUCTS_TOPIC, Consumed.with(stringSerde, stringSerde),
            Materialized.<String, String, KeyValueStore<Bytes, byte[]>>as("productsStore")
                    .withKeySerde(stringSerde)
                    .withValueSerde(stringSerde));

streamsBuilder.stream(CUSTOMERS_TOPIC, Consumed.with(stringSerde, stringSerde))
    // Key: customer ID, value: product ID
    .map((customerId, productId) -> KeyValue.pair(productId, String.format("%s|%s", customerId, productId)),
            Named.as("mapToProductIdAndCustomerIdProductId"))
    // Key: product ID, value: customer ID | product ID
    .join(productsTable, (customerIdProductId, productName) -> {
                String[] parts = customerIdProductId.split("\\|");
                String customerId = parts[0];
                return String.format("%s|%s", customerId, productName);
            },
            Joined.<String, String, String>as("customersProductsInnerJoin")
                    .withKeySerde(stringSerde)
                    .withValueSerde(stringSerde))
    // Key: product ID, value: customer ID | product name
    .map((productId, customerIdProductName) -> {
        String[] parts = customerIdProductName.split("\\|");
        String customerId = parts[0];
        String productName = parts[1];
        return KeyValue.pair(customerId, productName);
    }, Named.as("mapToCustomerIdAndProductName"))
    // Key: customer ID, value: product name
    .print(Printed.toSysOut());

Latest Value from the KStream for a given key

2 Answers2