Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
0
votes
1 answer

Python KafkaConsumer not printing output

I am using following script to read data from kafka - from kafka import KafkaConsumer from json import loads import json consumer = KafkaConsumer('topicname',bootstrap_servers=['brokerip'],enable_auto_commit=True) for message in consumer: …
Pratik das baghel
  • 155
  • 3
  • 10
0
votes
1 answer

Apache Kafka Unbalanced Cluster Logic

We have a requirement in which, we need to purposefully unbalanced the Kafka cluster by assigning the traffic distribution for a 3 broker cluster like 80 % to Broker 1 15 % to Broker 2 5 % to Broker 3 and send the messages for the topics to the…
Bhuvi
  • 13
  • 3
0
votes
0 answers

Apache Kafka Kerberos Authentication with Python

I need to develop a Python program which would act as a Kafka Consumer and do some processing based on that. I used kafka-python and it did the job fine with local testing. However, my production environment (RHEL7) requires Kerberos authentication.…
dab92
  • 155
  • 8
0
votes
1 answer

How to use Kafka with docker and python?

I am trying to set up a connection to Kafka using Docker. When running the code locally everything is working correctly, but when trying to run it using Docker Compose, I am getting the following error: Traceback (most recent call last): File…
Inon
  • 13
  • 3
0
votes
1 answer

Kafka-python producer.send isn't being recieved in a try-except block, but does send with time.sleep(1)

I'm testing a script that runs binwalk on a file and then sends a kafka message to let the sending file know that it was completed or if it failed. It looks like this: if __name__ == "__main__": # finds the path of this file scriptpath =…
Nicholas Raines
  • 1
0
votes
0 answers

AWS MSK working with shell command but not with python

I created a AWS MSK instance with two brokers. I can use that to produce and consume messages by using kafka#s shell commands (kafka-console-producer.sh and kafka-console-consumer.sh) but the problem is, it is not working with kafka python…
Mainul
  • 13
  • 5
0
votes
0 answers

kafka python not receiving message from golang producer

I am publishing messages via golang using segmentio-go. And I want to read those message using python for which I tried kafka-python, pykafka. In all these libraries, I cannot receive the message, I am able to read the messages when reading it from…
Mohammad Nayeem
  • 1
0
votes
1 answer

kafka-python: Can a consumer object join a consumer group without subscribing to any topics?

I have a topics_subscriber.py that continuously checks for any new topics and subscribes to them. def _subscribe_topics(consumer: KafkaConsumer) -> None: topics_to_subscribe = { topic for topic in consumer.topics() if…
rocky502
  • 23
  • 6
0
votes
1 answer

kafka python - What is the right way to find new kafka topics that a consumer has not been yet subscribed to?

I am new to the kafka world and trying to do the following for a kafka consumer in python get a list of all kafka topics. get a list of topics a consumer has subscribed to. subscribe to new topics (that have not yet been subscribed to). Note: I am…
rocky502
  • 23
  • 6
0
votes
1 answer

Limit the number of messages to consume in KafkaConsumer of kafka-python library

Sample code: consumer = KafkaConsumer(config["kafka"]["input"], bootstrap_servers=config["kafka"]["brokers"].split(','), value_deserializer=lambda m: json.loads(m.decode('ascii')), …
sandeep reddy
  • 11
  • 4
0
votes
1 answer

kafka-python: I am unable to run the example producer code

I have created Kafka cluster on a Google Cloud VM Firstly, I tested my broker using cli command to produce message: Producer: $ kafka-console-producer.sh --broker-list localhost:9092 --producer.config /opt/bitnami/kafka/conf/producer.properties…
Luke Mao
  • 87
  • 6
0
votes
0 answers

kafka-python KafkaProducer App failing (exiting with code 1) due to NameError: name 'json' is not defined

I am following this tutorial: https://florimond.dev/blog/articles/2018/09/building-a-streaming-fraud-detection-system-with-kafka-and-python/ The script to create the Producer App is failing, exiting with code 1, due to a NameError: name 'json' is…
user15037183
  • 47
  • 1
  • 5
0
votes
1 answer

Json producer in kafka-python

I am trying a simple kafka producer first time which will take data from json file record by record. But I am getting error. My Json File(test.json): { "states": [ { "name": "Alabama", "abbreviation": "AL" }, { …
Sun
  • 3,444
  • 7
  • 53
  • 83
0
votes
1 answer

Is there a way I can read large number of messages from a kafka topic using multiprocessing in python?

I am trying to read a kafka topic from the earliest offset and then tombstone certain records through a python script. Since the messages are huge in number (Million +), I want to leverage multiprocessing to make the script faster while consuming…
Rajat Bhardwaj
  • 129
  • 1
  • 11
0
votes
0 answers

Kafka producer with python multi threading

I'm planning to build a critical component which produces the messages to kafka. I'm just thinking, is there any way that python multithreading will help us in writing the efficient kafka producer with asyncio, multithreading ? Also here I'm…
Learnis
  • 526
  • 5
  • 25
1 2 3
…
29 30