0

Dependency used Alpakka Kafka 3.0

We have below consumer settings.

enable.auto.commit = true

auto.offset.reset = earliest

If We have enable.auto.commit = true then is it possible to consume messages from Kafka topic partition from a particular offset / from a date?

Akshay Jain
  • 73
  • 2
  • 8

1 Answers1

0

Use the assign method instead of subscribe method

 public void sample() {
        KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(properties);
        TopicPartition partition = new TopicPartition("some-topic", 0);
        consumer.assign(Arrays.asList(partition));
        consumer.seek(partition, 0);
        while (true) {
            final ConsumerRecords<Long, String> consumerRecords = consumer.poll(1000);
        }
    }

As far as I know, it is not necessary to specify the consumer group when you are using the assign method instead of subscribe method, in the Consumer properties, and hence probably will ignore those parameters (enable.auto.commit,auto.offset.reset). Use the assign method and seek only for debugging or testing purposes.

There is also another method called offsetsForTimes to get the desired offset to seek.

David Ogren
  • 4,396
  • 1
  • 19
  • 29
Felipe Tapia
  • 309
  • 2
  • 7
  • 1
    In Alpakka Kafka, you should be able to get similar results with the `plainExternalSource` in Alpakka's `Consumer`. – Levi Ramsey Aug 14 '22 at 21:47
  • Consumer offsets are stored in Kafka only. There is no other data source used for storing an offset. Is it possible to retrieve records from an offset or particular time stamp using the plainExternalSource API? – Akshay Jain Aug 15 '22 at 14:16
  • 1
    @AkshayJain, you would use the `MetadataClient` (see https://doc.akka.io/docs/alpakka-kafka/current/consumer-metadata.html), specifically its `.getCommittedOffsets` method to retrieve the committed offsets. Then based on the committed offsets (e.g. subtract 2000 from the last committed offsets in each partition), you would construct a `Subscription` to pass to the `plainExternalSource`. – Levi Ramsey Aug 15 '22 at 15:54
  • 1
    In Alpakka Kafka, after constructing a `KafkaConsumerActor`, you can get the offset for a particular time by using the ask pattern with a `GetOffsetsForTimes` message. – Levi Ramsey Aug 15 '22 at 15:56
  • @LeviRamsey , Can we use subscription API's ? For Partition Assignment with Offset we have Subscriptions.assignmentWithOffset API and For Partition Assignment with Timestamp we have Subscriptions.assignmentOffsetsForTimes API. Subscription API documentation [link](https://doc.akka.io/docs/alpakka-kafka/current/subscription.html) – Akshay Jain Aug 16 '22 at 07:50
  • 1
    Yeah, the Subscription API is what you'd use to tell the `plainExternalSource` what offset to start from. For getting a raw offset, you would use the `MetadataClient` and you shouldn't have to use anything special to get offsets for times. – Levi Ramsey Aug 16 '22 at 09:47
  • For the Subscriptions API, we need to pass TopicPartition. Is it possible to fetch the records from a particular timestamp without specifying partitions? – Akshay Jain Aug 17 '22 at 11:13