Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
2
votes
1 answer

Kafka on Docker for Windows - cannot consume data

I created a kafka setup using bitnami/kafka docker template. After recreating the containers multiple times (I wasn't able to connect to Kafka on 9092) I finally got my cluster up and running. I believe the problems were caused by the fact that…
BuahahaXD
  • 609
  • 2
  • 8
  • 24
2
votes
1 answer

Unable to import kafkaProducer for Syntax error in Library

I have installed the latest version Kafka using pip install Installing collected packages: kafka Successfully installed kafka-1.3.5 When I try to import KafkaProducer it gives me the error : File…
Sarang Manjrekar
  • 1,839
  • 5
  • 31
  • 61
2
votes
1 answer

Processing kafka messages taking long time

I have a Python process (or rather, set of processes running in parallel within a consumer group) that processes data according to inputs coming in as Kafka messages from certain topic. Usually each message is processed quickly, but sometimes,…
StasM
  • 10,593
  • 6
  • 56
  • 103
2
votes
1 answer

Handling a producer and consumer using Kafka-python

First I would like to say that I am a newbie to Kafka and also stackoverflow, So I am sorry if I am not asking this in the right way. I am trying to implement the producer-consumer using kafka-python. But its not working properly I have the…
Stan11
  • 274
  • 3
  • 11
2
votes
2 answers

How to reach kafka from within container on minikube?

I'm setting up a python app, and I would like to produce a message for kafka. The kafka server is running locally on my computer (not in minikube) and the app.py runs in a container in minikube. The code of app.py is this: try: producer =…
colo
  • 101
  • 8
2
votes
1 answer

Python, can't pass variable to KafkaConsumer

I am working on python application which connect to kafka and consume some data. def main(argv): params = parse_arg(argv) logging.info("Connecting to topic\t" + params.tasks_topic) consumer = KafkaConsumer(params.tasks_topic, …
Kuba Wenta
  • 580
  • 1
  • 5
  • 25
2
votes
1 answer

Is there a way to specify the isolation level in Kafka-Python consumer API?

I am trying to write a consumer using Kafka-Python ensuring the exactly once semantics. The messages in the partitions are produced using a transaction aware producer. I read from the Kafka docs that I should specify the isolation_level to be…
PythonLearner
  • 35
  • 6
2
votes
1 answer

Python Producer can send via shell, but not .py

I have a running and tested Kafka cluster, and am trying to use a Python script to send messages to the brokers. This works when I use the Python3 shell and call the producer method, however when I put these same commands into a python file and…
Tex
  • 89
  • 2
  • 10
2
votes
1 answer

Kafka-Python How to commit manually?

I am fairly new to Kafka and just started with Python, so pardon my wrongful terms, I'll try to explain my gist. I have a Kafka queue, where my topics are being generated pragmatically. Now these topics will hold a list of orders and will be picked…
Paras
  • 3,191
  • 6
  • 41
  • 77
2
votes
1 answer

How to create topics if it does not exists in Kafka dynamically using kafka-python

I'm fairly new to Python and just getting started with Kafka. I am using the library named python-kafka to communicate with Kafka. Now I have a requirement that I need to create topics dynamically, however if it does exists, I do not need to create…
Paras
  • 3,191
  • 6
  • 41
  • 77
2
votes
1 answer

Publish message to Confluent cloud cluster by kafka-python

I am using kafka-python package to publish messages to kafka confluent cloud cluster. My code looks like below: from kafka import KafkaProducer producer=KafkaProducer( …
Mahdi
  • 787
  • 1
  • 8
  • 33
2
votes
1 answer

Kafka Python producer integration with django web app

I have a question on how can we integrate kafka producer with a front end web app. get the data for every minute or second . Can the web app pass the JSON object to a running producer each time the it is created ? or do we need to initiate the kafka…
user2407164
  • 71
  • 5
2
votes
0 answers

Plotting live stream from Kafka-Python (with Plotly)

I am using kafka-python to create a Kafka consumer (connected to Kafka producer, broker etc.) in Python that is receiving a live stream of data. Now I just want to have a live timeseries plot of this data. I found Plotly as very nice Python library…
BobBetter
  • 195
  • 3
  • 14
2
votes
1 answer

Python kafka: Is there a way to block a consumer on a kafka topic till a new message is posted?

I have a consumer subscribed to a test topic where a producer thread posts regularly. I would like to be able to block the consumer thread until a new message is on - then process that and start waiting again. The closest I came is: consumer =…
Mr_and_Mrs_D
  • 32,208
  • 39
  • 178
  • 361
2
votes
0 answers

Kafka Python sending data with custom timestamp

I have JSON data with timestamp in it like this a = {'key1':'value','timestamp':'123344453'} I am using kafka python version 1.3.4.1 My producer code: producer = KafkaProducer( bootstrap_servers='localhost:9092', value_serializer=lambda v:…
Dev_Man
  • 847
  • 1
  • 10
  • 28
1 2 3
…
29 30