1

Currently, my Kafka Consumer streaming application is manually committing the offsets into Kafka with enable.auto.commit set to false. The application failed when I tried restarting it throwing below exception:

org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions:{partition-12=155555555}

Assuming the above error is due to the message not present/partition deleted due to retention period, I tried below method:

I disabled the manual commit and enabled auto commit(enable.auto.commit=true and auto.offset.reset=earliest) Still it fails with the same error

org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions:{partition-12=155555555}

Please suggest ways to restart the job so that it can successfully read the correct offset for which message/partition is present

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
user1326784
  • 627
  • 3
  • 11
  • 31
  • possible [duplicate](https://stackoverflow.com/questions/37320643/kafka-consumer-offsets-out-of-range-with-no-configured-reset-policy-for-partitio) – senseiwu Feb 26 '19 at 11:06

2 Answers2

0

You are trying to read offset 155555555 from partition 12 of topic partition, but -most probably- it might have already been deleted due to your retention policy.

You can either use Kafka Streams Application Reset Tool in order to reset your Kafka Streams application's internal state, such that it can reprocess its input data from scratch

$ bin/kafka-streams-application-reset.sh

Option (* = required)         Description
---------------------         -----------
* --application-id <id>       The Kafka Streams application ID (application.id)
--bootstrap-servers <urls>    Comma-separated list of broker urls with format: HOST1:PORT1,HOST2:PORT2
                                (default: localhost:9092)
--intermediate-topics <list>  Comma-separated list of intermediate user topics
--input-topics <list>         Comma-separated list of user input topics
--zookeeper <url>             Format: HOST:POST
                                (default: localhost:2181)

or start your consumer using a fresh consumer group ID.

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
  • 1
    Using these options will work. But I do not understand why the `auto.offset.reset` strategy is not applied. – Val Bonn Feb 27 '19 at 07:50
  • @ValBonn This configuration is applied only when there are no committed offsets to the group. This is why I mentioned the use of a fresh group ID. – Giorgos Myrianthous Feb 27 '19 at 07:57
  • Thank you @Giorgos , but it is not the way I understand the documentation https://kafka.apache.org/documentation/ : `auto.offset.reset ` : `What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):` And @user1326784 seems to be in this second case. – Val Bonn Feb 27 '19 at 08:06
  • @ValBonn I guess that @user1326784 should also try with `auto.offset.reset=largest` and see what happens - in order to make sure that reset policy is not effective in this scenario. – Giorgos Myrianthous Feb 27 '19 at 09:18
-1

I met the same problem and I use package org.apache.spark.streaming.kafka010 in my application.In the begining,I suscepted the auto.offset.reset strategy take no effect,but when I read the description of the method fixKafkaParams in the object KafkaUtils,i found the configuration has been overwrited.I guess the reason why it tweak the configuration ConsumerConfig.AUTO_OFFSET_RESET_CONFIG for executor is to keep consistent offset obtained by driver and executor.

suohala
  • 1
  • 1