I want to try the kafka-python package instead of the Confluent Kafka python binding because I want a pure Python package and only kafka-python does that. However, I am unable to work out how to implement a particular behaviour that I depend on in Confluent Kafka.
In the Confluent Kafka python binding it is possible to configure the consumer to return an error message when the end of a partition has been reached. I use this to detect when there are no more messages to process, at least for a while. However, I cannot work out the right way to do the same thing with kafka-python.
I could request the end offset from the KakfaConsumer before I start processing messages but this introduces a race condition; the end offset could move after I query it and while I'm processing messages. Alternatively, I could request end offsets after processing every message but I'm worried that there may be a performance penalty associated with this approach.
So, while I have a number of ideas about what I could do, I don't know what I should do.
Any clues as to the accepted method would be welcome.
