1

I am using Kafka Transactional producer to post atomically to 2 topics on a broker. My code looks similar to this:

Properties props = new Properties();
 props.put("bootstrap.servers", "localhost:9092");
 props.put("transactional.id", "my-transactional-id");
 Producer<String, String> producer = new KafkaProducer<>(props, new StringSerializer(), new StringSerializer());

 producer.initTransactions();

 try {
     producer.beginTransaction();
     for (int i = 0; i < 100; i++)
         producer.send(new ProducerRecord<>("my-topic", Integer.toString(i), Integer.toString(i)));
     producer.commitTransaction();
 } catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
     // We can't recover from these exceptions, so our only option is to close the producer and exit.
     producer.close();
 } catch (KafkaException e) {
     // For all other exceptions, just abort the transaction and try again.
     producer.abortTransaction();
 }
 producer.close();

Acc. to kafka docs and my understanding, initTransactions() has to be called only once per producer session, and it registers that producer instance to the broker with the specified transactional-id.

Now, in my case I need to deploy this code to several servers using same kafka broker. Do I need different transactional Ids for each instance? Is there any way to close initTransactions() once it is called so that it doesn't block other producers executing transactions as they have same transactional-id.

P.s. I don't want to close the producer and re-instantiate it after every sent transaction as this can impact performance, I believe. How can we implement an efficient solution to this problem?

Mehul
  • 47
  • 5
  • 1
    _To two topics_ - but this code only uses one `"my-topic"`? Use the server hostname or other unique identifier appended to the transaction ID if you are trying to make it different in each server – OneCricketeer Dec 18 '21 at 15:14

0 Answers0