how to write to many output types from one input type?

Question

I am a newbie to kafka streams.

I am able to achieve the following. I have an input topic. The message object (say Flower). I have a transformer that converts Flower into Fruit. I then store the Fruit to output topic.

What I want to achieve is like something that follows

I store Fruit to Fruit topic (... as above)
I store (the successful) Flower to Successful_Flower topic (... additionally)

(If it is not possible to discriminate Flowers as above, it is even okay if I can store the incoming Flower (successful or failed ...both) to OutputFlower topic)

Is this possible? If yes, how? Please let me know if I need to supply more details. Thanks

score 0 · Answer 1 · answered Dec 06 '20 at 19:14

For this you can use the method branch as follow:

    KStream<String, Either<Flower, Fruit>>[] flowerAndFruitStreams = inputMessages
            .flatMapValues(flower -> List.<Either<Flower, Fruit>>of(Either.left(flower), Either.right(fromFlowerToFruit(flower)))) 
            .branch((key, value) -> value.isLeft(), (key, value) -> value.isRight());

    KStream<String, Either<Flower, Fruit>> flowerStream = flowerAndFruitStreams[0];
    KStream<String, Either<Flower, Fruit>> fruitsStream = flowerAndFruitStreams[1];

Explanations

As you can see, the branch method returns a KStream<K, V>[] which means that the input is split in many streams based on the predicates applied to this method. Because of this, after you convert a flower to fruit you should return an object which either contains the flower, either the fruit so that to have a stream of the same objects type. For this we will borrow from functional programming the concept of Either which you can find it below:

import java.util.Optional;
import java.util.function.Consumer;
import java.util.function.Function;

public class Either<L, R> {

    private final Optional<L> left;
    private final Optional<R> right;

    private Either(Optional<L> l, Optional<R> r) {
        left = l;
        right = r;
    }

    public static <L, R> Either<L, R> left(L value) {
        return new Either<>(Optional.of(value), Optional.empty());
    }

    public static <L, R> Either<L, R> right(R value) {
        return new Either<>(Optional.empty(), Optional.of(value));
    }

    public boolean isLeft() {
        return left.isPresent();
    }

    public boolean isRight() {
        return right.isPresent();
    }

    public L getLeft() {
        return left.get();
    }

    public R getRight() {
        return right.get();
    }

    public <T> T map(Function<? super L, ? extends T> leftFunc, Function<? super R, ? extends T> rightFunc) {
        return left.<T>map(leftFunc).orElseGet(() -> right.map(rightFunc).get());
    }

    public <T> Either<T, R> mapLeft(Function<? super L, ? extends T> leftFunc) {
        return new Either<>(left.map(leftFunc), right);
    }

    public <T> Either<L, T> mapRight(Function<? super R, ? extends T> rightFunc) {
        return new Either<>(left, right.map(rightFunc));
    }

    public void apply(Consumer<? super L> leftFunc, Consumer<? super R> rightFunc) {
        left.ifPresent(leftFunc);
        right.ifPresent(rightFunc);
    }
}

I didn't compiled the code, so please double check the syntax. This code is based on Java 13 and KafkaStreams 2.5.0 but should work also for newest versions.

how to write to many output types from one input type?

1 Answers1