2

Good afternoon. Given: a kafka topic with more than 20 partitions and a huge message traffic and one consumer group. It is necessary each time connecting to a topic to reset all the lag available there (with data loss) on all partitions and read it from the most recent messages. Please tell me how to implement it. I use python + confluent-kafka. I need something like this pseudocode.

But it has two problems, I don't know how to get the list of partitions correctly and this code throws an error when trying to read from a topic.

def reset_offset_to_latest(consumer, partitions):
    for p in partitions:
        p.offset = confluent_kafka.OFFSET_END
    print('assign', partitions)
    consumer.assign(partitions)

Error:

cimpl.KafkaException: KafkaError{code=_STATE,val=-172,str="Failed to set assignment: Local: Erroneous state"}

Please help!

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Valtesar
  • 41
  • 4
  • Your code has indentation problems. Code inside a function needs to be indented. Try to paste your code into the interpreter. It will raise a SyntaxError. Though most likely this isn't what caused the exception. Since SyntaxError will prevent the code from running in the first place. But your code format in the question is wrong. – Ξένη Γήινος Jul 19 '23 at 11:25
  • when inserting the code, the indents moved out – Valtesar Jul 19 '23 at 11:33
  • Don't do extra indentation, just indent the code as you would normally in the console. Use three backticks before the code block and after it. – Ξένη Γήινος Jul 19 '23 at 11:35
  • I don't understand your pseudocode. There already is a seek_to_end function... And the assign API doesn't use consumer groups, only subscribe does – OneCricketeer Jul 19 '23 at 13:08

0 Answers0