Stack Overflow
Stack Overflow

Questions tagged [kafka-transactions-api]

The Kafka Transactions API allows writing to and reading from topics as atomic operations, enabling the implementation of exactly-once strategies.

The Kafka Transactions API allows writing to and reading from topics as atomic operations, enabling the implementation of exactly-once strategies. Exactly-Once means that each message read is processed exactly once and, as a rule, exactly one message is written to the output topic.

33 questions
8
votes
1 answer

What are Kafka transactions?

What do transactions mean in Kafka? Of course I know the ordinary SQL transactions: A transaction is a sequence of operations performed (using one or more SQL statements) on a database as a single logical unit of work So does it mean it's possible…
J.J. Beam
  • 2,612
  • 2
  • 26
  • 55
5
votes
1 answer

How to choose Kafka transactional.id in a Kubernetes (Producer side only transaction) set up

I have a Kafka wrapper library that uses transactions on the produce side only. The library does not cover the consumer. The producer publishes to multiple topics. The goal is to achieve transactionality. So the produce should either succeed which…
user14205224
  • 53
  • 2
4
votes
1 answer

When to use Kafka transactional API?

I was trying to understand Kafka's transactional API. This link defines atomic read-process-write cycle as follows: First, let’s consider what an atomic read-process-write cycle means. In a nutshell, it means that if an application consumes a…
Mahesha999
  • 22,693
  • 29
  • 116
  • 189
3
votes
1 answer

Getting ProducerFencedException on producing record in Kafka listener thread

I am getting this exception when producing the message inside the kafka listener container. javax.management.InstanceAlreadyExistsException:…
Faizan
  • 289
  • 4
  • 15
3
votes
1 answer

Kafka setting consumer offset before committed offset

I'm running a program that starts with a message in the topic, consumes it, processes it, commits the next offset, and publishes a new message to the same topic, all in a transactional fashion. I have the following (simplified) trace: Fetch…
gurghet
  • 7,591
  • 4
  • 36
  • 63
2
votes
0 answers

Can't initialize transactions

I have local instances of Kafka and Zookeeper running in Docker on my local machine (macOS Monterey) which have worked quite well for my needs so far. I recently want to implement something with Kafka transactions and run into the problem that I…
clemens
  • 16,716
  • 11
  • 50
  • 65
2
votes
2 answers

How a consumer will read committed messages?

So, as far as I understand from Transactions in Apache Kafka, a read_committed consumer will not return the messages which are part of an ongoing transaction. So, I guess, the consumer will have the option to commit its offset past those…
Adrian
  • 3,321
  • 2
  • 29
  • 46
2
votes
0 answers

Kafka transaction: Receiving CONCURRENT_TRANSACTIONS on AddPartitionsToTxnRequest

I am trying to publish in a transaction a message on 16 Kafka partitions on 7 brokers. The flow is like this: open transaction write a message to 16 partitions commit transaction sleep 25 ms repeat Sometimes the transaction takes over 1 second to…
edxvshacks
  • 21
  • 3
1
vote
0 answers

Is it possible to re-read a message in Kafka Streams?

I'm processing a message from a topic in a Kafka stream in a transaction. Based on message processing logic I would like to be able to either move forward with a next message from the topic, or re-consume the currently processed message (sort of…
AndCode
  • 384
  • 1
  • 10
1
vote
1 answer

transaction-id-prefix need to be identical for same application instance in spring kafka 3?

I understand in spring kafka 3, I need a unique transaction-id-prefix for applications running with multiple instances. here is my configuration: fun getTxnIdPrefix() { return "$hostname-mytransaction-id-prefix-${UUID.randomUUID()}" } in this…
jon
  • 395
  • 1
  • 3
  • 21
1
vote
0 answers

Kafka Transaction INVALID_PRODUCR_EPOCH Error while Commit / Abort Transaction

I am facing the INVALID_PRODUCER_EPOCH error while committing or aborting the transaction. Am running the my application in Windows 10 machine. I have a single Producer and TransactionId is Random Guid, enabled Idempotence in producer as well. No…
Vishnu Kumar K S D
  • 21
  • 4
1
vote
1 answer

Kafka transactions and Producer buffering mechanism

I need to get exactly-once semantics, so I use Kafka Transactional API. And I'm trying to understand how to work with Producer efficiently. As I read in some articles, it's more optimized way to use only one Producer per application instance (within…
Singularis24
  • 11
  • 2
1
vote
0 answers

Apache Kafka transactional producer does not honour atomicity while posting to 2 topics, if one topic goes down

I am using Kafka Transactional producer to post atomically to 2 topics on a broker. My code looks similar to this: Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("transactional.id",…
Mehul
  • 47
  • 5
1
vote
0 answers

How to run several instances of a kafka transactional producer on same broker with same transactionalId?

I am using Kafka Transactional producer to post atomically to 2 topics on a broker. My code looks similar to this: Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("transactional.id",…
Mehul
  • 47
  • 5
1
vote
0 answers

Is it possible to force abort a Kafka transaction?

We have a test Kafka cluster that we were experimenting with adjusting various settings. One of the settings that was adjusted was to set the transaction.max.timeout.ms to 7 days. While that setting was in place we had a network failure to one of…
Karl
  • 23
  • 1
  • 5
1
2 3