0

For transactions that have been started by a listener container, we need to set the same transaction id prefix for all app instances. For producing-only transaction, we need to set a different value for each instance instead.

I'm using Spring Cloud Stream Kafka Binder in the app which has both types of transactions, and the property spring.cloud.stream.kafka.binder.transaction.transaction-id-prefix is used for creating a common transaction manager.

I wonder how to make it all work properly because it seems that you can't have it both ways at the same time.

amseager
  • 5,795
  • 4
  • 24
  • 47

1 Answers1

2

That requirement is no longer necessary if your broker is 2.5 or later and you are using a currently supported version of spring-cloud-stream (3.2.x) and spring-kafka (2.8.x).

You must now use a unique prefix even for consumer-initiated transactions.

If you are using an older broker or an unsupported spring-kafka version that does not use EOSMode V2 (BETA) by default, you can configure two binders.

https://docs.spring.io/spring-kafka/docs/current/reference/html/#transaction-id-prefix

This problem (different rules for transactional.id) has been eliminated when EOSMode.BETA is being used (with broker versions >= 2.5); see Exactly Once Semantics.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Thanks, apparently missed this part of docs. So, it looks like it should be ok to set any random string (like random uuid) to this property for each app instance, right? – amseager Jun 29 '22 at 15:29
  • Yes, that is correct. – Gary Russell Jun 29 '22 at 15:47
  • 1
    No; if they both send records using the same producer; there is only need for one prefix. – Gary Russell May 24 '23 at 13:02
  • accidentally deleted my comment, gary's above answer is for this: after spring 3, in 1 application instance, I have 10 consumers listening to 10 topics, do I need different transaction-id-prefix for them? e.g. `myhostA-topicId-partitionId` – jon May 24 '23 at 14:56
  • 1
    But it must be unique to each instance, if you run multiple app instances. – Gary Russell May 24 '23 at 14:58