1

I am using Kafka Transactional producer to post atomically to 2 topics on a broker. My code looks similar to this:

 Properties props = new Properties();
 props.put("bootstrap.servers", "localhost:9092");
 props.put("transactional.id", "my-transactional-id");
 Producer<String, String> producer = new KafkaProducer<>(props, new StringSerializer(), new StringSerializer());
 
 ProducerRecord<String, String> record1 = new ProducerRecord("topic-1", null, (Object) null, payload, headerList);
 ProducerRecord<String, String> record2 = new ProducerRecord("topic-2", null, (Object) null, payload, headerList);
 List<ProducerRecord<String, String>> recordList = Arrays.asList(record1, record2);

 producer.initTransactions();

 try {
     producer.beginTransaction();
     Iterator var2 = recordList.iterator();

     while(var2.hasNext()) {
         ProducerRecord<K, V> record = (ProducerRecord)var2.next();
         this.send(record, (Callback)null);
     }
     
     producer.commitTransaction();
 } catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
     // We can't recover from these exceptions, so our only option is to close the producer and exit.
     producer.close();
 } catch (KafkaException e) {
     // For all other exceptions, just abort the transaction and try again.
     producer.abortTransaction();
 }
 producer.close();

Now, in order to test the atomicity while posting to both the topics, I deleted "topic-2". I am expecting the transaction to fail completely. But strangely after several retries it commits transaction successfully to "topic-1".

Also, I am seeing continuous error logs with messages:

Error while fetching metadata with correlation id 123 : {topic-2=UNKNOWN_TOPIC_OR_PARTITION}

But eventually it says

Transition from state IN_TRANSACTION to COMMITTING_TRANSACTION

and then posts successfully to "topic-1". I am not sure why am I seeing this behaviour. What would possibly go wrong and is this behaviour expected?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Mehul
  • 47
  • 5
  • Why wouldn't it be expected? A non existing topic is not a fatal error and transactions are only bounded by one topic. If you want to atomically post to multiple topics, you'll probably need to chain callbacks – OneCricketeer Jan 05 '22 at 13:11

0 Answers0