Reading last Kafka partition's record that's transactionally written

Question

I'm trying to read the last record of a topic's partition. The topic's producer is writing transactionally. The consumer is set up with isolation.level=read_committed. I manage the offsets manually. Here's my consumer code:

// Fetch the end offset of a partition
Map<TopicPartition, Long> endOffsets = consumer.endOffsets(Collections.singleton(topicPartition));
Long endOffset = endOffsets.get(topicPartition);

// Try to read the last record
if (endOffset > 0) {
    consumer.seek(topicPartition, Math.max(0, endOffset - 5));
    List records = consumer.poll(10 * 1000).records(topicPartition);

    if (records.isEmpty()) {
        throw new IllegalStateException("How can this be?");
    } else {
        // Return the last record
        return records.get(records.size() - 1);
    }
}

So to read the last record I ask for the end offset, then seek to endOffset - 5 (because of Kafka skipping offsets in exactly-once mode, like I've seen in this question: Kafka Streams does not increment offset by 1 when producing to topic), and start polling.

It's been working well before, but now I've got an exception telling me zero records are polled. What can be a reason for that? I can't reproduce it and I think I'm lost.

Are you sure there is a message in the topic? – Amin Mar 07 '19 at 08:27 — Amin, Mar 07 '19 at 08:27
Tens of thousands of them. – Alexander Eliseyev Mar 07 '19 at 08:29 — Alexander Eliseyev, Mar 07 '19 at 08:29
Did you change `endOffset - 5` to any greater number? – Amin Mar 07 '19 at 08:42 — Amin, Mar 07 '19 at 08:42

Reading last Kafka partition's record that's transactionally written

0 Answers0