spring-kafka how to set Retry on the stream bean

Question

I saw spring-kafka supports Non-Blocking Retries using @RetryableTopic. I only saw @RetryableTopic is working with @kafkaListener together. But I want the "Retry" on my stream aggregation. How to do that by spring-kafka?

The code example below is about bank transactions (stream) and account balances (state).
Say a bank transaction is like this: move $10 from account 10001 to account 10002.
I have the stream code below using reduce function to -10 from 10001 and +10 to 10002.
And the balance is materialized to state store BALANCE.
if the account 10001 balance is less than 10, the transaction shall not be fulfilled. But it shall be retried, because a deposit transaction may come in a short period. And after the deposit to 10001, the balance is > 10, then this transaction shall be fulfilled.

Here is my stream bean

    @Bean
    public KStream<String, BankTransaction> alphaBankKStream(StreamsBuilder streamsBuilder) {
        JsonSerde<BankTransaction> valueSerde = new JsonSerde<>(BankTransaction.class);
        KStream<String, BankTransaction> stream = streamsBuilder.stream(Topic.TRANSACTION_RAW,
                Consumed.with(Serdes.String(), valueSerde));

        KStream<String, BankTransaction>[] branches = stream.branch(
                (key, value) -> isBalanceEnough(value),
                (key, value) -> true                 /* all other records  */
        );

        branches[0].flatMap((k, v) -> {
            List<BankTransactionInternal> txInternals = BankTransactionInternal.splitBankTransaction(v);
            List<KeyValue<String, BankTransactionInternal>> result = new LinkedList<>();
            result.add(KeyValue.pair(v.getFromAccount(), txInternals.get(0)));
            result.add(KeyValue.pair(v.getToAccount(), txInternals.get(1)));
            return result;
        }).filter((k, v) -> !Constants.EXTERNAL_ACCOUNT.equalsIgnoreCase(k))
                .map((k,v) -> KeyValue.pair(k, v.getAmount()))
                .groupBy((account, amount) -> account, Grouped.with(Serdes.String(), Serdes.Double()))
                .reduce(Double::sum,
                        Materialized.<String, Double, KeyValueStore<Bytes, byte[]>>as(StateStore.BALANCE).withValueSerde(Serdes.Double()));

        return stream;
    }

    private boolean isBalanceEnough(BankTransaction bankTransaction) {
        // read balance from state store BALANCE
        return balance >= bankTransaction.amount
    }

score 0 · Answer 1 · answered May 03 '21 at 13:06

KStream is outside the scope of Spring for Apache Kafka; spring is only involved with setting up the topology; once it it set up, you are using kafka-streams directly. All the functionality is provided by the topology you set up.

The @RetrybleTopic feature only applies to @KafkaListener (or more specifically the kafka lister containers).

spring-kafka how to set Retry on the stream bean

1 Answers1