I am using Kafka Transactional producer to post atomically to 2 topics on a broker. My code looks similar to this:
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("transactional.id", "my-transactional-id");
Producer<String, String> producer = new KafkaProducer<>(props, new StringSerializer(), new StringSerializer());
ProducerRecord<String, String> record1 = new ProducerRecord("topic-1", null, (Object) null, payload, headerList);
ProducerRecord<String, String> record2 = new ProducerRecord("topic-2", null, (Object) null, payload, headerList);
List<ProducerRecord<String, String>> recordList = Arrays.asList(record1, record2);
producer.initTransactions();
try {
producer.beginTransaction();
Iterator var2 = recordList.iterator();
while(var2.hasNext()) {
ProducerRecord<K, V> record = (ProducerRecord)var2.next();
this.send(record, (Callback)null);
}
producer.commitTransaction();
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
// We can't recover from these exceptions, so our only option is to close the producer and exit.
producer.close();
} catch (KafkaException e) {
// For all other exceptions, just abort the transaction and try again.
producer.abortTransaction();
}
producer.close();
Now, in order to test the atomicity while posting to both the topics, I deleted "topic-2". I am expecting the transaction to fail completely. But strangely after several retries it commits transaction successfully to "topic-1".
Also, I am seeing continuous error logs with messages:
Error while fetching metadata with correlation id 123 : {topic-2=UNKNOWN_TOPIC_OR_PARTITION}
But eventually it says
Transition from state IN_TRANSACTION to COMMITTING_TRANSACTION
and then posts successfully to "topic-1". I am not sure why am I seeing this behaviour. What would possibly go wrong and is this behaviour expected?