Kafka Streams: Partial reprocessing by key

Question

Scenario:

In a KafkaStreams web sessioning scenario, with unlimited (or years-long) retention, with interactive queries (this can be reviewed if necessary), with many clients, which have many users each (each user particular to each client), and where partitioning goes like this:

Partition by a function of (clientId, userId) % numberOfPartitions, setting a numberOfPartitions beforehand depending on the cluster size. This would allow sessioning to be performed on (clientId,userId) data, and should provide an even data distribution among the nodes (no hotspotting, on partition size or on write load).

However, when querying, I'd query by client(and time range). So then, I'd build an aggregated Ktable from that Sessions table, where key is the client, and Sessions are queried by (client, timeStart, timeEnd). That would make that data from a client to have to go into one node, which could pose scalability issues(too big a client), but since data is aggregated already, I guess that would be manageable.

Question:

In this scenario (variants appreciated), I'd like to be able to reprocess only for one client.

But data from one client would be scattered among (potentially all of) the partitions.

How can a partial reprocess be achieved in Kafka Streams with minor impact, and keep (old) state queryable in the meantime?

Answer 1

I think in general you already know the answer to your question, with the partitioning scheme like you have described it you will have to read all partitions if you want to reprocess a client, as the messages will be spread throughout all of them.

The only thing that I can come up with to limit the amount of overhead when reprocessing an entire client is to implement a partitioning scheme that groups several partitions for a client and then distributes users over those partitions to avoid overloading one partition with a particularly "large" client. The picture should hopefully clarify what I am probably failing to explain with words..

A custom partitioner to achieve this distribution could look somewhat like the following code. Please take this with a grain of salt, this is purely theoretical so far and has never been tested (or even run for that matter) - but it should illustrate the principle.

public class ClientUserPartitioner implements Partitioner {
int partitionGroupSize = 10;

@Override
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
    // For this we expect the key to be of the format "client/user"
    String[] splitValues = ((String)key).split("/");
    String client = splitValues[0];
    String user = splitValues[1];

    // Check that partitioncount is divisible by group size
    if (cluster.availablePartitionsForTopic(topic).size() % partitionGroupSize != 0) {
        throw new ConfigException("Partitioncount must be divisible by "+ partitionGroupSize +" for this partitioner but is " +
                cluster.availablePartitionsForTopic(topic).size() + " for topic " + topic);
    }

    // Calculate partition group from client and specific partition from user
    int clientPartitionOffset = Utils.murmur2(client.getBytes()) % partitionGroupSize * partitionGroupSize;
    int userPartition = Utils.murmur2(user.getBytes()) % partitionGroupSize;

    // Combine group and specific value to get final partition
    return clientPartitionOffset + userPartition;
}

@Override
public void configure(Map<String, ?> configs) {
    if (configs.containsKey("partition.group.size")) {
        this.partitionGroupSize = Integer.parseInt((String)configs.get("partition.group.size"));
    }
}

@Override
public void close() {
}

}

Doing this will of course impact your distribution, it might be worth your while to run a few simulations with different values for partitionGroupSize and a representative sample of your data to estimate how even the distribution is and how much overhead you'd save when reprocessing an entire client.