0

I need to do a programmatic commit using kafka-python api. From the the documentation i see that I need to use the api below, but couldn't find a good usage example:

enter image description here

It needs offsets as a dictionary of TopicPartition and OffsetMetadata. I was thinking of using the assignment() API on the consumer which returns me the list of TopicPartitions of the current consumer. For the OffsetAndMetadata, i can set the offset but I don't get what i need to set for the metadata attribute. The documentation isn't that helpful.

Please let me know.

Thanks in advance.

hakish
  • 3,990
  • 7
  • 39
  • 56
  • If Java API is any indication, the `metadata` is optional - https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html. Also, this SO may help https://stackoverflow.com/questions/36579815/kafka-python-how-do-i-commit-a-partition? – mazaneicha Feb 21 '20 at 18:51

1 Answers1

1

Metadata is optional in OffsetAndMetadata object. You can pass None as the metadata value.

Kafka does not use this metadata internally so it is effectively an opaque byte array that can be used to store application-specific data.

More info here: https://github.com/dpkp/kafka-python/issues/645

Arthur Erlendsson
  • 176
  • 3