10

I am new to Kafka Streams, I am using version 1.0.0. I would like to set a new key for a KTable from one of the values.

When using KStream, it cane be done by using method selectKey() like this.

kstream.selectKey ((k,v) -> v.newKey)

However such method is missing in KTable. Only way is to convert given KTable to KStream. Any thoughts on this issue? Its changing a key against design of KTable?

Stefan Repcek
  • 2,553
  • 4
  • 21
  • 29

5 Answers5

22

If you want to set a new key, you need to re-group the KTable:

KTable newTable = table.groupBy(/*put select key function here*/)
                       .aggregate(...);

Because a key must be unique for a KTable (in contrast to a KStream) it's required to specify an aggregation function that aggregates all records with same (new) key into a single value.

Since Kafka 2.5, Kafka Streams also support KStream#toTable() operator. Thus, it is also possible to do table.toStream().selectKey(...).toTable(). There are advantages and disadvantages for both approaches.

The main disadvantage of using toTable() is that it will repartition the input data based on the new key, which leads to interleaves writes into the repartition topic and thus to out-of-order data. While the first approach via groupBy() uses the same implementation, using the aggregation function helps you to resolve "conflicts" expliclity. If you use the toTable() operator, an "blind" upsert based on offset order of the repartition topic is done (this is actually similar to the code example in the other answers).

Example:

Key | Value
 A  | (a,1)
 B  | (a,2)

If you re-key on a your output table would be either once of both (but it's not defined with one):

Key | Value          Key | Value
 a  | 1               a  |  2

The operation to "rekey" a table is semantically always ill-defined.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • 1
    can you chck my answer please? I don't know the reasoning behind Kafka Streams API design, but it sounds logical based on the way Kafka Streams are parallelized. – yuranos Mar 30 '21 at 22:50
  • 1
    What you say is correct. Note though, that you could "miss-use" the aggregation step, by just applying an aggregator `(k,v,a) -> v` that just blindly picks the "latest" value -- this "latest" value will we in offset order of the repartition topic (that `groupBy()` implies), but of course the order will be "non-deterministic" for the reason you laid out... – Matthias J. Sax Mar 31 '21 at 01:33
  • Some other proposed answers unfortunately follow this pattern, but it's actually broken... (ie, non-deterministic...) – Matthias J. Sax Mar 31 '21 at 01:37
  • @MatthiasJ.Sax So which pattern actually to use? How can I determine which message is the latest for a particular key? And do I get it right, that after the first repartitioning to a KTable you cannot guarantee KTable semantics anymore? – fachexot Sep 21 '21 at 13:32
  • 1
    It depends on your use-case and requirements. For example, if you know that no two records from different input partitions cannot map to the same key, using `selectKey().toTable()` is just fine. (Similar, if both records are "far apart" from each other, as in practice no race-condition would occur.) -- Using the aggregation approach has the advantage that you compare the current and new value for a key, and make a decision to apply or not apply the update (for example, you could compare the timestamp of the messages). – Matthias J. Sax Sep 21 '21 at 23:40
  • After a repartitioning, we cannot guarantee _ordering_ any longer. Only if there are potential race-conditions, this may become an issue. -- We are also working on some other improvements to be able to handle out-of-order data for KTables better out-of-the-box. – Matthias J. Sax Sep 21 '21 at 23:41
  • Is there any new method or improvement for re-keying KTables? Does it work deterministically, if all messages of one partition result again in the same partition after the groupby? – fachexot May 19 '22 at 14:39
  • `Is there any new method or improvement for re-keying KTables?` -- no. If you want to resolve conflicts, you can try to use `aggregate` (you can compare current and new row for this case -- you might to add some metadata to the value). -- `Does it work deterministically, if all messages of one partition result again in the same partition after the groupby?` I think yes. – Matthias J. Sax May 19 '22 at 17:00
11

@Matthias's answer led me down the right path, but I thought having a sample piece of code might help out here

final KTable<String, User> usersKeyedByApplicationIDKTable = usersKTable.groupBy(
        // First, going to set the new key to the user's application id
        (userId, user) -> KeyValue.pair(user.getApplicationID().toString(), user)
).aggregate(
        // Initiate the aggregate value
        () -> null,
        // adder (doing nothing, just passing the user through as the value)
        (applicationId, user, aggValue) -> user,
        // subtractor (doing nothing, just passing the user through as the value)
        (applicationId, user, aggValue) -> user
);

KGroupedTable aggregate() documentation: https://kafka.apache.org/20/javadoc/org/apache/kafka/streams/kstream/KGroupedTable.html#aggregate-org.apache.kafka.streams.kstream.Initializer-org.apache.kafka.streams.kstream.Aggregator-org.apache.kafka.streams.kstream.Aggregator-org.apache.kafka.streams.kstream.Materialized-

Allen Underwood
  • 517
  • 5
  • 11
  • 1
    on subtract shouldn't you return null? – Tudor Mar 31 '20 at 08:23
  • 2
    The program you write is non-determinism though... The same issue applies to @Jackson Oliveira approach: if you have two upstream record that map to the same new key, you have no idea which one of both will end up on the table... – Matthias J. Sax Mar 31 '21 at 01:36
4

I don't think the way @Matthias described it is accurate/detailed enough. It is correct, but the root cause of such limitation(exists for ksqlDB CREATE TABLE syntax as well) is beyond just sheer fact that the keys must be unique for KTable.

The uniqueness in itself doesn't limit KTables. After all, any underlying topic can, and often does, contain messages with the same keys. KTable has no problem with that. It will just enforce the latest state for each key. There are multiple consequences of this, including the fact that KTable built from aggregated function can produce several messages into its output topic based on a single input message...But let's get back to your question.

So, the KTable needs to know which message for a specific key is the last message, meaning it's the latest state for the key.

What ordering guarantees does Kafka have? Correct, on per partition basis.

What happens when messages are re-keyed? Correct, they will be spread across partitions very different from the input message.

So, the initial messages with the same key were correctly stored by the broker itself into the same partition(if you didn't do anything fancy/stupid with your custom Partitioner) That way KTable can always infer the latest state.

But what happens if the messages are re-keyed inside Kafka Streams application in-flight?

They will spread across partitions again, but with a different key now, and if your application is scaled out and you have several tasks working in parallel you simply can't guarantee that the last message by a new key is actually the last message as it was stored in the original topic. Separate tasks don't have any coordination like that. And they can't. It won't be efficient otherwise.

As a result, KTable will lose its main semantic if such re-keying were allowed.

yuranos
  • 8,799
  • 9
  • 56
  • 65
  • "They will spread across partitions again, but with a different key now" What exactly do you mean by that? That key is now different from original (but all messages with same original key have same new key) or keys from messages with same original key differ now? – fachexot Sep 22 '21 at 09:21
  • 1
    "but all messages with same original key have same new key" - that's not the case if you are doing re-keying. The name of the operation speaks for itself. Imagine you might have had a message={user: fachexot} and a key for it key=123. Depending what you want to do, maybe it makes sense to use a user name as a key instead, add something else to themessage body and even discard the original surrogate key altogether. So, you might get message={lastActive: 2021-09-22}, key=fachexot as a result. – yuranos Sep 22 '21 at 19:38
  • I think it can very well be the case: e.g. message1={user: fachexot, number: 1} with key=123 and message2={user: fachexot, number: 2} with key=123. After Re-Keying: message1={user: fachexot, number: 1} with key=fachexot and message2={user: fachexot, number: 2} with key=fachexot – fachexot Sep 23 '21 at 13:41
  • 1
    Your example has nothing to do with my original statement. "They will spread across partitions again, but with a different key now". In your example it's exactly what happens because the original key was 123 and as a result of the transformation it became fachexot. But even if the idea is just to show that two messages can still end up in the same partition, it's just a specific example. It can be that way or it can be very different. It's not about a specific example, but about overall Kafka Streams design. – yuranos Sep 23 '21 at 19:43
  • I have to ask again for my case: Can order be guaranteed in my example above? When two messages with same key on same partition result in two messages with new but again same key on same partition -> can i re-key like that without getting out of order? – fachexot May 19 '22 at 14:36
  • 1
    You wouldn't get an out-of-order issue for those two events specifically. But some other events that were stored in source Kafka topics before or after the original events can arrive at target topics out of order depending on the overall complexity of your transformations. – yuranos May 20 '22 at 09:39
3

For the ones who are using confluent 5.5.+ there is a method that allows extract the key from a stream and convert to a KTable directly:

       KTable<String, User> userTable = builder
            .stream("topic_name", Consumed.with(userIdSerde, userSerde))
            .selectKey((key, value) -> key.getUserId())             
            .toTable( Materialized.with(stringIdSerde, userSerde));

Details can be found here

Jackson Oliveira
  • 61
  • 1
  • 5
    Well, while this "works" there is one important thing to consider. If there are two row in the input table, that map to the same new key, there is no guarantee in which order both records will arrive in the result table. Thus, it might be a non-deterministic program. – Matthias J. Sax Sep 25 '20 at 22:33
1

@Allen Underwood code helped me, had to make some changes if key is custom Pojo. As i was getting class cast exception. Below code worked

usersKTable.groupBy((k, v) -> KeyValue.pair(v.getCompositeKey(), v),Grouped.with(compositeKeySerde,valueSerde))
                .aggregate(
                        () -> null,
                        (applicationId, value, aggValue) -> value,
                        (applicationId, value, aggValue) -> value,
                        Materialized.with(compositeKeySerde, valueSerde)
                );
Sumeet
  • 23
  • 2
  • 1
    Found another easy way, not sure if efficient though. Convert table to stream and then using select key change the key. Push that stream to new topic and then let table read from new topic. – Sumeet Feb 05 '20 at 10:56
  • 1
    The program you write is non-determinism though... The same issue applies to @Jackson Oliveira approach: if you have two upstream record that map to the same new key, you have no idea which one of both will end up on the table.... – Matthias J. Sax Mar 31 '21 at 01:35