Recently started working with Kafka in an offline environment. Tools used are: kafka-python (2.0.2), kafka_2.12-2.8.1 and zookeeper 3.7.0, running everything on LAN network of machines using Windows 10.
I can't share the code, because offline, but the details are listed below:
The kafkaConsumer reads a message that contains a network path to a "text-based file"(ex: .txt, .pdf, .docx, etc.) and runs them through a tika docker container to get the text via pycurl. This process takes anywhere from 30 seconds to 1.5 minutes based on the length of the text-based file that is sent (these are averages). My problem is no matter what I change, either the max_poll_size or max_poll_interval_ms, even though all the files take about the same time to be completed, the worker group occasionally decides to rebalance and create this dreaded error... - even with an absurdly high max_poll_interval_ms, the max I have tried is 720000 ms (12 min) and I have tried ranges from 100-1000 for max_poll_request. With this continuing, I then tried to manually commit the messages once the they were all completed w/ the following code:
offset = consumer.committed(TopicPartition(topic,partition))
options = {TopicPartition(topic,partition): OffsetAndMetadata(offset+len(list_of_message_values), partion)
consumer.commit(options)
This produced the same issue though... but I did figure out a "hack" work around...
Instead of subscribing the topic to the KafkaConsumer, I instead had to manually assign the Consumer w/ a TopicPartition(topic, partition) w/ the assign method instead.
This is not very pythonic, and additionally because I have 5 partitions this means 5 scripts - which is just annoying and difficult to make sure the code is the same through out and if one partition finishes before the other, it won't utilize the other consumer. Is there something else I can try that is proper and not a "hack" like this? Thanks.
