Stack Overflow
Stack Overflow

Questions tagged [pykafka]

PyKafka is a cluster-aware Kafka>=0.8.2 client for Python

About

PyKafka is a cluster-aware Kafka>=0.8.2 client for Python. It includes Python implementations of Kafka producers and consumers, which are optionally backed by a C extension built on librdkafka, and runs under Python 2.7+, Python 3.4+, and PyPy.

Links

55 questions
0
votes
0 answers

Read specific record with pykafka

I want to store big files in Kafka, using metadata about the record to retrieve them in the future. So I send around messages containing the topic, partition_id, offset and then I try to retrieve the file in this way: def…
Chobeat
  • 3,445
  • 6
  • 41
  • 59
0
votes
0 answers

pykafka:Socket disconnected during offset manager discovery

This is my code： enter image description here ​When I run it,it will warn me: Socket disconnected during offset manager discovery So I use Ctrl+C to stop it Then enter image description here I need your help ,what should I do? Thank you very much!
yixius
  • 3
  • 2
0
votes
1 answer

How kafka recognizes consumer group?

I have this code to create consumer balanced in pykafka : consumer = topic.get_balanced_consumer(consumer_group='Testing',auto_commit_enable=True,zookeeper_connect='amsmgmt002:2181,nas5:2181,amsdblx006:2181') How the state of consumer is…
quantCode
  • 495
  • 1
  • 5
  • 12
0
votes
1 answer

Async not working when using pykafka and asyncio

I try to call multiple pykafka consumer function using async. However, the first pykafka consumer function will block the other function from working. The QueueConsumer lib: import json from pykafka import KafkaClient import configparser import…
muhammad septiadi
  • 61
  • 1
  • 4
0
votes
1 answer

How to use pykafka group consumer with gevent?

I use pykafka group consumer with gevent, but the results have repeating data. show my code: import gevent from pykafka import KafkaClient topic_name = 'test2' bootstrap_servers =…
eagle A
  • 107
  • 8
0
votes
2 answers

pykafka topics return None

from pykafka import KafkaClient client = KafkaClient(hosts='192.168.199.87:9092') topics = client.topics print(topics) the result of topics is a dict with all values are None: {b'user_name_topic': None, b'test_topic': None} how to fix it?
eagle A
  • 107
  • 8
0
votes
0 answers

How to know the allocation of kafka partition

I'm using kafka 0.9.0 with pykafka as a url queue in scrapy, and the topic i used have 30 partitions with 4 consumers. It works properly at first, but when it worked about a couple of hours the kafka seems having a problem with rebalancing the…
jason
  • 11
  • 4
0
votes
1 answer

PyKafka fails to connect to partitioned kafka topic

apache-storm-1.0.2 hbase-1.2.1 kafka_2.10-0.10.0.0 zookeeper-3.4.9 ops: zkServer.sh start $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &! $KAFKA_HOME/bin/kafka-topics.sh --zookeeper localhost:2181 --create…
DmitrySemenov
  • 9,204
  • 15
  • 76
  • 121
0
votes
0 answers

Producer can not connect to broker through DNS

I have a physical server, where i have set the advertised.host.name to server ip, and do the port forwarding at the router. But the producer cannot connect to broker by using the dns. ERROR:pykafka.connection:Failed to connect to…
Yong
  • 1
  • 4
-1
votes
1 answer

pykafka consumes same messages

Pykafka Kafkaconsumer consumes same messages from beginning everytime. def consume_from_kafka(): client = KafkaClient(hosts=IP) topic = client.topics[TOPIC] consumer =…
newUser
  • 386
  • 5
  • 17
1 2 3
4