0

I use spring cloud kafka stream to do some processing to the stream save in the middle of the topology some data to the db and then continue the process in my topology and finally send final result to other topic

My Function bean looks something like this:

@Component("handler")
public Class Handler implements Function<KStrean<Key1,Value1>,KStream<Key2,Valu2>> {
KStream<Key2,Value2> apply(KStream<Key1,Value1> input){
//start process topology and make few transformation
input.mapValues(...).filter...(...)
//now save the the intermediate result to DB via Spring data JPA and Hibernate
someEntityJpaRepo.save()
//continue with the remaning topology

Where should I use the @Transactional annotation if I want all or nothing behavior like in RDBMS and what PlatformTransactionalManager should I use? Jpa? Kafka? For example I want to in the case of exception in the DB query or exception in the stream processing that the db transaction will roll back as well the kafa transaction

user1409534
  • 2,140
  • 4
  • 27
  • 33

1 Answers1

1

KafkaTransactionManager is NOT for KStream - kafka streams manage their own transactions.

Since Spring is not involved with Kafka Streams at runtime (only for topology setup), there is no concept of transaction synchronization between the two.

The JPA transaction will be independent of the Kafka Streams transaction and you must deal with the possibility of duplicate deliveries (make the DB update idempotent or use some other technique to detect that this Kafka record has already been stored to the DB.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Can't I use exactly once semantics in kafka stream to avoid duplicate deliveries even in case of failure in RDBMS query failure? – user1409534 Aug 17 '22 at 03:40
  • 1
    It's a common misunderstanding; "exactly once" means the entire `consume->process->produce` sequence will be completed exactly once. If any error occurs after the DB commits and before the kafka transaction is committed, the `consume->process` part can be called again (that part is "at least once"). – Gary Russell Aug 17 '22 at 14:07