python kafka for consuming does not work

Question

I can write to kafka. However, consuming does not work

kafka = KafkaClient(kafka_host_list)
consumer = SimpleConsumer(kafka,'topic-test-development','topic-test-development')

No handlers could be found for logger "kafka.conn"
Traceback (most recent call last):
  File "/home/ubuntu/workspace/feed-tests/tests/kafka_consumer.py", line 44, in <module>
    consumer = SimpleConsumer(kafka,'topic-test-development','topic-test-development')
  File "/usr/local/lib/python2.7/dist-packages/kafka/consumer/simple.py", line 126, in __init__
    auto_commit_every_t=auto_commit_every_t)
  File "/usr/local/lib/python2.7/dist-packages/kafka/consumer/base.py", line 75, in __init__
    self.fetch_last_known_offsets(partitions)
  File "/usr/local/lib/python2.7/dist-packages/kafka/consumer/base.py", line 101, in fetch_last_known_offsets
    check_error(resp)
  File "/usr/local/lib/python2.7/dist-packages/kafka/common.py", line 230, in check_error
    raise response
kafka.common.FailedPayloadsError

What is the version of python kafka client? There is introduction on the doc: group: a name for this consumer, used for offset storage and must be unique If you are connecting to a server that does not support offset commit/fetch (any prior to 0.8.1.1), then you *must* set this to None (FROM:https://github.com/mumrah/kafka-python/blob/adbd4ac052e4a5b40cfc2a3589b7adbcb656afe5/kafka/consumer/simple.py) — BAE, Aug 06 '15 at 20:19
My experience with Kafka and python-kafka wrapper is the broker connection is flaky, meaning the connection can be reset when producer tries to send a message or when consumer tries to mark a message as consumed. Your best bet is to implement a retry logic because I don't think this problem is getting fixed anytime soon. I'm still seeing them pop up here and there. — adam, Oct 12 '15 at 13:06
I too had the same error when my client was on a different machine from the broker... i went really deep into python-kafka code and found out, the broker publishes its name as `localhost` and the client when fetches the broker hostname metadata as localhost, it tries to make connection to itself... I fixed it by setting `advertised.host.name` to external hostname in `server.properties` — , Apr 28 '16 at 16:23

score 3 · Answer 1 · answered Jul 13 '15 at 07:41

note No handlers could be found for logger "kafka.conn"

add code for logger:

logging.basicConfig(
    format='%(asctime)s.%(msecs)s:%(name)s:%(thread)d:%(levelname)s:%(process)d:%(message)s',
    level=logging.DEBUG
)

run and output more info:

2015-07-13 15:26:32,158.158.830881119:kafka.conn:140247909795584:DEBUG:17035:Reinitializing socket connection for host01:9092 2015-07-13 15:26:32,221.221.35591507:kafka.conn:140247909795584:ERROR:17035:Unable to connect to kafka broker at host01:9092

python kafka for consuming does not work

1 Answers1