7

I am using the transactional KafkaProducer to send messages to a topic. This works fine. I use a KafkaConsumer with read_committed isolation level and I have an issue with the seek and seekToEnd methods. According to the documentation, the seek and seekToEnd methods give me the LSO (Last Stable Offset). But this is a bit confusing. As it gives me always the same value, the END of the topic. No matter if the last entry is committed (by the Producer) or part of an aborted transaction. Example, after I abort the last 5 tries to insert 20_000 messages, the last 100_000 records should not be read by the Consumer. But during a seekToEnd it moves to the end of the Topic (including the 100_000 messages). But the poll() does not return them.

I am looking for a way to retrieve the Last Committed Offset (so the last successful committed message by the Producer). There seems to be no proper API method for this. So do I need to roll my own?

Option would be to move back and poll until no more records are retrieved, this would result in the last committed message. But I would assume that Kafka provides this method.

We use Kafka 1.0.0.

Coen Damen
  • 2,009
  • 5
  • 29
  • 51
  • Can you provide your full configuration? Also, can you try to `seek` to `-3`? `-3` is a sentinel value representing the `last stable offset`. – salient Jan 23 '18 at 11:48

2 Answers2

1

The class KafkaConsumer has some nice methods like: partitionFor, begginingOffsets and endOffsets also commited and position.

Check which one fits to your needs. Especially carefully consider all 4 offset-related methods. The method partitionFor returns complete metadata object with other information, but can be useful for enriching the logging.

Seweryn Habdank-Wojewódzki
  • 2,037
  • 4
  • 30
  • 51
  • 1
    I have raised a JIRA @ Kafka. Seek and SeekToEnd, endOffsets etc move to the last message on the Topic, regardless of the message being part of a committed transaction or not.... – Coen Damen Jan 26 '18 at 08:13
  • Actually we need something like seekToLastCommittedMessage – JR ibkr Mar 22 '19 at 20:13
  • Can you specify how did you get the last successfully committed message by the Producer ? As per my understanding, seekToEnd() and endOffsets() will take you to the end of your topic. In my case, I have 100 message in a topic. Offset#96 was the last committed message by a producer. Offset#96-100 are uncommitted messages. How do I retrieve last committed message ? – JR ibkr Mar 25 '19 at 16:41
  • @CoenDamen can you please link the kafka JIRA? thanks! – roman-roman Mar 27 '19 at 17:45
0

To get the last committed offset of a topic partitions you can use the KafkaConsumer.committed(TopicPartition partition) function. 

TopicPartition topicPartition = new TopicPartition(record.topic(), record.partition());
Long committedOffset = consumer.committed(topicPartition).offset();
System.out.println("last committed offset: " + committedOffset);
jjbskir
  • 8,474
  • 9
  • 40
  • 53