Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
1
vote
1 answer

Issue in rewinding to oldest available message for KafkaConsumer

I have rewinded fetch offset for one Kafka consumer in group G1 to off1 by following code: consumer1.set_topic_partitions((topic, partition, off1)) In the above line, off1 is offset of oldest available message for the topic in corresponding…
Joy
  • 4,197
  • 14
  • 61
  • 131
0
votes
1 answer

How to send a message to every consumer who is idle in Kafka?

I have three consumers and one producer in Kafka. When the producer sends all the messages (there are 100 messages in my simple code), these messages are divided among three consumers, and my main problem is this division of messages. Sometimes a…
Sardar
  • 524
  • 1
  • 6
  • 19
0
votes
0 answers

Data stream with confluent-kafka python giving error - TypeError: You must pass either str or Schema

I am trying to build Stream-Table Join Operations in Kafka python and below is the code that performs join operation on data sent from producer stream_table_join.py from confluent_kafka import DeserializingConsumer, SerializingProducer from…
Polymath
  • 328
  • 1
  • 9
0
votes
1 answer

Batch process Kafka messages

I want to write my Kafka messages to a jsonl file which should each contain a number of lines (let's say 2). My producer currently writes 3 messages at a time so I should get two jsonl files: one with 2 events and a second one with 1 event. My…
Omega
  • 750
  • 1
  • 8
  • 24
0
votes
0 answers

Can´t Produce to topic in Kafka using docker and python

So I'm facing an odd situation where I can connect to my kafka broker but can´t produce to the topic I created. I have three images running on same network: zookeeper,kafka and my app. I have a simple producer in my app that goes like this: import…
Pedro Silva
  • 1
  • 1
0
votes
0 answers

No data from the Kafka Consumer Python - Consumer listens but nothing is consumed

I'm searching for a solution to use Kafka to display my API (localhost) to my docker. My producer file works fine and here it is Consumer file on the other hand returns this... I tried creating a new test container in powershell to debug, gave me…
Hawvb Ged
  • 1
0
votes
0 answers

Is change kafka to kafka-python in docker container is changing from docker-compose configuration?

Previosly I have error like this SyntaxError on "self.async" when running python kafka producer Based on the answer, best answer is switch from kafka package to kafka-python package, this is what the detail of my docker-compose configuration, I…
Nabih Bawazir
  • 6,381
  • 7
  • 37
  • 70
0
votes
0 answers

Python Kafka Comsumer.poll gives error when using gzipped avro format

I have a line in my code which gives an immediate error when trying to consume messages that are both avro format and gzipped. message = consumer.poll(300000) Here is the consumer config .. consumer = KafkaConsumer( bootstrap_servers=broker, …
mez63
  • 166
  • 4
  • 20
0
votes
1 answer

Why do I get a ValueError in my Kafka Consumer if I seek to another position?

I'm using python 3.9.16 and kafka-python version 2.0.2. I'm running on my Macbook Pro IOS 11.6.5. I'm still getting my feet wet with Kafka so it's entirely possible I'm doing things the wrong way. What I'm trying to do is test seeking to offsets…
Classified
  • 5,759
  • 18
  • 68
  • 99
0
votes
0 answers

Publish a Message with schema id with kafka-python

I am using kafka-python and I publish to a strimzi broker with schema registry an avro serialized message. The schema is already registered on the broker. My code looks like the following: schema = avro.schema.parse(open("schema.avsc",…
user3579222
  • 1,103
  • 11
  • 28
0
votes
0 answers

kafka.consumer.fetcher.ConsumerRecord access to topic name

I'm trying to process a message received from akafka consumer for message in self.consumer: try: self.process_message(message) except Exception as e: print(f"Error: {str(e)}") When I try to read topic name…
Paolo Colombo
  • 1
  • 1
0
votes
1 answer

How does Kafka store messages offsets on a local computer?

How does Kafka store messages on a local server or laptop? I'm new to Kafka and just playing around with the tech for now but I'm curious to the answer because I started by looking at the Kafka Quickstart page to just start a Kafka service on my…
Classified
  • 5,759
  • 18
  • 68
  • 99
0
votes
1 answer

ModuleNotFoundError: No module named 'kafka'

I have downloaded Kafka and installed kafka-python library using "pip install kafka-python" and "conda install -c conda-forge kafka-python". I am able to run "from kafka import KafkaProducer" inside a python shell on the terminal. But the same line…
SHIVANI SINGH
  • 25
  • 5
0
votes
0 answers

How to setup Python connection to Kafka instance running in Kubernetes?

I am trying to connect to Kafka using Python and I am able to connect and see the topic names however when I try to read messages for any topic nothing happens. It feels like the connection times-out as python does not throw an error. I am…
Prabhat_Seth
  • 1
  • 2
0
votes
0 answers

kafka docker python API does not recognizes the broker

I have a kafka consumer in VSCode IDE (MS Windows) that works properly. I take the same python script in docker container, but the kafka broker is not recognized. I put them inside the same network; so networking is OK. What could be the…
FSH
  • 1
1 2 3
…
29 30