2

I use Apache Kafka client to poll records from Kafka.

Let us say I do something like:

org.apache.kafka.clients.consumer.Consumer consumer = ...
...
ConsumerRecords<String, String> consumerRecords = consumer.poll(...);
List<ConsumerRecord<String, String>> records = consumerRecords.records(partition);

My question: are records returned in the natural order? More specifically, are they ordered by offset in the records list?

I couldn't find anything specific about this property in the JavaDoc.

Andremoniy
  • 34,031
  • 20
  • 135
  • 241

2 Answers2

2

Looking at the source code, it seems that, yes, the order of records inside a partition is kept. The records are stored in an ArrayList and fetched using offsets https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java

See also the answers at Apache Kafka order of messages with multiple partitions

Anton Malyshev
  • 8,686
  • 2
  • 27
  • 45
1

Polling implies that the records are coming back asynchronously. You aren't guaranteed any order, except the order in which they were processed.

If you need to sort the List, you can use any Comparable function you wish to sort the stream once it's complete.

duffymo
  • 305,152
  • 44
  • 369
  • 561
  • It's my understanding of what asynchronous, event based processing means and adding events to streams. What needs strengthening? Kafka is an event streaming bus that different applications can connect to for messaging. There's no guarantee about anyone who connects to the message bus. You cannot know when they will emit an event. You have no idea what they are all doing or how long their process will take. – duffymo Jan 17 '18 at 12:49
  • I would suggest that you improve your question. – duffymo Jan 17 '18 at 12:51
  • if there is only one partition, is polling order guaranteed? As per the linked shared in above answer, it does. – Govinda Sakhare Apr 08 '20 at 03:24