Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
0
votes
1 answer

How to reach Cloudera Kafka Broker on private network from outside?

I have a cluster inside a VPN which contains a server with private IP. I'm trying to set up a Kafka communication between an external server to my private server. My approach is to set an IP table where a public IP is pointing my private IP. Also, I…
Marc
  • 17
  • 4
0
votes
1 answer

Apache Kafka: Programmatic monitoring of consumer groups

I would like to monitor the consumer groups (Kafka based groups not ZK) in my Kafka cluster from within a Python application (I am open to using either of the Kafka-python library and the confluent Kafka client). The purpose is to launch more (or…
void
  • 2,403
  • 6
  • 28
  • 53
0
votes
1 answer

Pushing messages in kafka to separate partitions by a key

I have a Kafka topic (test-topic) with 3 partitions, and a set of messages which contain a key that can take only 3 types of values, I want these messages to go to separate partitions based on their value. from kafka import KafkaProducer from…
Devaraj Phukan
  • 196
  • 1
  • 9
0
votes
2 answers

Specify SSL details for Kafka brokers in python

I need to connect to kafka instance which has multiple brokers with SSL. I am using kafka-python to consume and process data. I have seen link where they used kafka-python to connect to single broker with SSL. Link :…
Lijo Jose
  • 317
  • 1
  • 3
  • 13
0
votes
1 answer

can not use kafka for python

python:2.6.6 kafka-python:1.4.3 i had run the kafka producer,but it always tips me this error: Traceback (most recent call last): File "KafkaOperation.py", line 11, in from kafka import KafkaProducer File…
Carol Jack
  • 3
  • 2
0
votes
1 answer

Unable to publish in Kafka

I want to publish in Kafka topic I am unable to do so, the program halts. I am getting this error: KafkaTimeoutError: Failed to update metadata after 60.0 secs. def saveResults(response): entities_tweet = response["entities"] for entity in…
hasherBaba
  • 319
  • 5
  • 18
0
votes
1 answer

KafkaUnavailableError in python app

I am new to Kafka and following this tutorial to go through a simple python app. When I run the producer.py, I get below error raise KafkaUnavailableError('All servers failed to process request: %s' % hosts) kafka.errors.KafkaUnavailableError:…
nad
  • 2,640
  • 11
  • 55
  • 96
0
votes
1 answer

How to kafka-python consumers parallely

I have a set of kafka-python consumers that consume from different kafka topics continuously and parallely. My question is how to kick off the consumers in parallel using single python script? And what is the best way to manage(start/stop/monitor)…
yoga
  • 1
0
votes
1 answer

Why does adding more consumers not increase the throughput?

I have a single node Kafka setup on a VM with 56GB RAM and 750GB disk with one broker. This is the what the server.properties file looks…
void
  • 2,403
  • 6
  • 28
  • 53
0
votes
2 answers

kafka-python producer not able to send when run in a docker container

I am using kafka-python (pip install kafka-python) in a Flask application to send messages to a Kafka cluster (running version 0.11). The application is deployed to AWS elastic beanstalk via docker. However, I see no messages reaching Kafka…
Nik
  • 5,515
  • 14
  • 49
  • 75
0
votes
0 answers

Access Kafka data in instance variable from Flask endpoints

I have written a Kafka consumer, in which I am capturing some metrics inside a class variable, below is the code of consumer. class Consumer(Thread): def __init__(self, kafka_topic, kafka_brokers, group_id): Thread.__init__(self) …
Sumit Kumar
  • 402
  • 1
  • 6
  • 26
0
votes
1 answer

Kafka-python connection lost

I would like to know how to verify the connection to kafka: let me to explain better: I have one KafkaConsumer is waiting to consume messages, let's assume that kafka connection meantime is down. While I am trying to consume messages with my…
Prisco
  • 653
  • 1
  • 15
  • 30
0
votes
0 answers

how to improve python-kafka consume throughout

I test my python-kafka script, the kafka-producer produced 10000/s message, but I consumed about 1000/s, so how to improve my consumer throughout? while True: topic_to_records = consumer.poll(100) if len(topic_to_records.values()) == 0: …
lizx
  • 21
  • 1
  • 5
0
votes
2 answers

Multiple consumers with same group id in Python

Does anybody know how to run multiple consumers with same group id in Python? I've tried following a = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'dd1', 'default.topic.config': {'auto.offset.reset': 'smallest'}}) b =…
arturo_bandini
  • 21
  • 1
  • 3
0
votes
1 answer

Python-Kafka : Program running in interactive mode and not in script mode

I am trying to send some messages to a Kafka topic in Machine_2 through a python script in Machine_1. Both Machine_2 and Machine_1 are in the same network and both are VM's in Azure. Code: sampl.py from kafka import KafkaProducer Producer =…
wonder
  • 885
  • 1
  • 18
  • 32
1 2 3
…
29 30