Stack Overflow
Stack Overflow

Questions tagged [kafka-transactions-api]

The Kafka Transactions API allows writing to and reading from topics as atomic operations, enabling the implementation of exactly-once strategies.

The Kafka Transactions API allows writing to and reading from topics as atomic operations, enabling the implementation of exactly-once strategies. Exactly-Once means that each message read is processed exactly once and, as a rule, exactly one message is written to the output topic.

33 questions
1
vote
0 answers

Kafka transactions how to emulate an error to rollback transaction

I want to test Kafka transactional behavior in depth. I need to test what happens in the transaction boundary if this is a possibility. For example, I want the commit of the transaction to fail so that I can test what happens during a transaction…
alex
  • 705
  • 9
  • 21
1
vote
1 answer

Consumer does not read messages from the Kafka topic (Akka Stream Kafka)

We use Akka Stream Kafka for producing and consuming messages and Strimzi Kafka cluster. Here are the versions if matters: com.typesafe.akka:akka-stream-kafka_2.13:2.0.7 com.typesafe.akka:akka-stream_2.13:2.6.14 …
Barat Sahdzijeu
  • 1,683
  • 1
  • 18
  • 29
1
vote
1 answer

Kafka Transactional read committed Consumer

I have transactional and normal Producer in application which are writting to topic kafka-topic as below. Configuration for transactional Kafka Producer @Bean public Map producerConfigs() { Map props =…
Nalini Wanjale
  • 1,717
  • 4
  • 19
  • 24
0
votes
0 answers

Should I recreate producer after ProducerFencedException?

If I got ProducerFencedException in my transactional producer, should I try to recreate it? If I got this exception, it means that there is another producer with the same transaction.id, but if a new producer is closed (replica is down), my old…
hololensen
  • 171
  • 9
0
votes
1 answer

Can I consume and produce multiple times in a single Kafka transaction?

I'm studying kafka transaction and have a question. I understood a Kafka transaction to be a bundle of one consume and one produce. Then, can I consume and produce multiple times in a single Kafka transaction? For example, application A produces…
harvey24
  • 3
  • 2
0
votes
0 answers

In transactional kafka, is it possile to find out if an offset is an actual message or a control record (a transaction commit/abort) marker message?

In my project, I seek to a particular offset and then do consumer.poll In case the offset is a control record, the poll method returns zero records. I want to know if it is because the offset has a control record or is it because of some other error…
vaibhav
  • 3
  • 2
0
votes
1 answer

Why my Spring Kafka unit test almost ran into ProducerFencedException every time

The test is consist of 2 test cases @Test @Nested inner class Test1 { fun test1() { mypublisher.publishInTransaction(topic1) // see log1 // check listener1 // see log2 } } @Test @Nested inner class Test2 { fun test2() { …
jon
  • 395
  • 1
  • 3
  • 21
0
votes
1 answer

Transactional.id in a spring transactional kafka producer

In my application, I have producers publishing to a kafka topic (with only 1 partition) & multiple consumers (each consumer is in its own consumer group) consuming from the topic. Now, for some reason I need to use the transactional kafka producer…
vaibhav
  • 3
  • 2
0
votes
1 answer

Will the order of received messages between consumers be preserved during the transaction in Kafka?

Usually, the order of messages between Consumers is not guaranteed, since this prohibits CAP-theorem. Consistency in this theorem can be determined through the theory of Actors, that is, when we preserve the fact of receiving messages and their…
skyho
  • 1,438
  • 2
  • 20
  • 47
0
votes
1 answer

Invalid transition attempted from state COMMITTING_TRANSACTION to state ABORTING_TRANSACTION in Producer

I am trying to achieve exactly once functionality but getting KafkaException with message as "org.apache.kafka.common.KafkaException: TransactionalId db13196c-6974-48b0-9835-aed40cec4ca4: Invalid transition attempted from state…
Harshit Vijayvargia
  • 173
  • 2
  • 12
0
votes
2 answers

Transaction with Fs2-kafka

I am looking for a way to use producer with transaction, using fs2, however the current TransactionalProducer seems to be geared toward a scenario in which it is an end to end workflow, meaning consume-process-produce. However, we would like to use…
MaatDeamon
  • 9,532
  • 9
  • 60
  • 127
0
votes
1 answer

Unable to push subsequent transactions to Kafka Producer

I'm trying to send events to Kafka in batches (let's say 10 events in a batch) to maintain transactional guarantees. But I'm able to successfully send only the first batch of events to Kafka. All subsequent batches fail with the reason…
Shivam
  • 11
  • 3
0
votes
0 answers

Topic authorization failed

We are using Transactional concept of Kafka where we consume the message from one topic and process it and produce the message on another topic. Our cluster has security enabled, so we have provided write operation to transaction-id using ACL. But…
Smith
  • 151
  • 3
  • 11
0
votes
1 answer

Kafka Message recovery with AfterRollbackProcessor for Non retry exception

With respect to the link - Kafka transaction rollback not working with 3 topics for RecordTooLargeException On EDIT3 I have below question how to send error to DB and at the same time to DLQ while using AfterRollbackProcessor I added …
user3575226
  • 111
  • 5
  • 15
0
votes
1 answer

Kafka transaction rollback not working with 3 topics for RecordTooLargeException

I post 3 message to 3 topics - while posting if gets exception - all messages will be rolled back. But in my case it is not happening when I simulate the below exception for 3rd Topic. org.apache.kafka.common.errors.RecordTooLargeException: The…
user3575226
  • 111
  • 5
  • 15
1
2
3