Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
12
votes
3 answers

Python: how to mock a kafka topic for unit tests?

We have a message scheduler that generates a hash-key from the message attributes before placing it on a Kafka topic queue with the key. This is done for de-duplication purposes. However, I am not sure how I could possibly test this deduplication…
user1658296
  • 1,398
  • 2
  • 18
  • 46
12
votes
6 answers

kafka-python - How do I commit a partition?

Using kafka-python-1.0.2. If I have a topic with 10 partitions, how do I go about committing a particular partition, while looping through the various partitions and messages. I just cant seem find an example of this anywhere, in the docs or…
Johnny Gasyna
  • 461
  • 1
  • 3
  • 13
12
votes
1 answer

Kafka python consumer reading all the messages when started

I am using the below code to read messages from a topic. I am facing two issues. Whenever i start consumer, it is reading all the messages in the queue? How do read only the unread messages? from kafka import KafkaConsumer consumer =…
user3570620
  • 359
  • 1
  • 6
  • 16
11
votes
3 answers

How to pass data from Kafka to Spark Streaming?

I am trying to pass data from kafka to spark streaming. This is what I've done till now: Installed both kafka and spark Started zookeeper with default properties config Started kafka server with default properties config Started kafka…
HackCode
  • 1,837
  • 6
  • 35
  • 66
10
votes
1 answer

KafkaTimeoutError: Failed to update metadata after 60.0 secs

I have a use case of high throughput kafka producer where I want to push thousands of json messages every second. I have a 3 node kafka cluster and I am using latest kafka-python library and have following method to produce message def…
Sarvesh Kumar
  • 131
  • 1
  • 1
  • 4
10
votes
0 answers

UnknownMemberId Error in AIOKafka library while consuming messages

I am facing an error with the library AIOKafka in Python (versions at the end). Basically, I am receiving a failed heartbeat message and then the commit of the offsets cannot be performed. This is the log: Heartbeat failed for group…
Victor Cadena
  • 191
  • 2
  • 7
10
votes
1 answer

kafka-python consumer start reading from offset (automatically)

I'm trying to build an application with kafka-python where a consumer reads data from a range of topics. It is extremely important that the consumer never reads the same message twice, but also never misses a message. Everything seems to be working…
Steven
  • 619
  • 1
  • 7
  • 21
9
votes
3 answers

Can't Consume JSON Messages From Kafka Using Kafka-Python's Deserializer

I am trying to send a very simple JSON object through Kafka and read it out the other side using Python and kafka-python. However, I keep seeing the following error: 2017-04-07 10:28:52,030.30.9998989105:kafka.future:8228:ERROR:10620:Error…
jencoston
  • 1,262
  • 7
  • 19
  • 35
9
votes
5 answers

Kafka Consumer: How to start consuming from the last message in Python

I am using Kafka 0.8.1 and Kafka python-0.9.0. In my setup, I have 2 kafka brokers setup. When I run my kafka consumer, I can see it retrieving messages from the queue and keeping track of offsets for both the brokers. Everything works great! My…
user1797335
  • 99
  • 1
  • 1
  • 3
8
votes
1 answer

Kafka python graceful shutdown of consumer

I am trying to gracefully shutdown a kafka consumer, but the script blocks with Stopping HeartBeat thread. How can i gracefully close the consumer on a SIGTERM with kafka-python. This is what i have done import logger as logging import time import…
Savitha Suresh
  • 321
  • 4
  • 12
8
votes
1 answer

How to programmatically check if Kafka Broker is up and running in Python

I'm trying to consume messages from a Kafka topic. I'm using a wrapper around confluent_kafka consumer. I need to check if connection is established before I start consuming messages. I read that the consumer is lazy, so I need to perform some…
ghost
  • 1,107
  • 3
  • 12
  • 31
8
votes
3 answers

How to force a consumer to read a specific partition in kafka

I have an application for downloading specific web-content, from a stream of URL's generated from 1 Kafka-producer. I've created a topic with 5 partitions and there are 5 kafka-consumers. However the timeout for the webpage download is 60…
ashdnik
  • 648
  • 3
  • 8
  • 20
8
votes
3 answers

Kafka produce.send never sends the message

I am using Kafka 2.12 and kafka-python module as Kafka client. I am trying to test a simple producer: class Producer(Process): daemon = True def run(self): producer = KafkaProducer(bootstrap_servers='kafka:9092') print("Sending…
Ivan
  • 323
  • 2
  • 3
  • 11
8
votes
1 answer

No Brokers Available error when trying to connect to Kafka

I have a very strange problem when trying to connect locally to Kafka 0.10.0.0 using Python client on CentOS. My connection options are pretty simple and default: kafka_consumer = kafka.KafkaConsumer( bootstrap_servers=['localhost:9092'], …
Andrey Lifanov
  • 81
  • 1
  • 1
  • 2
8
votes
1 answer

multiprocessing in kafka-python

I have been using the python-kaka module to consume from a kafka broker. I want to consume from the same topic with 'x' number of partitions in parallel. The documentation has this : # Use multiple consumers in parallel w/ 0.9 kafka brokers #…
red_devil
  • 1,009
  • 2
  • 13
  • 23
1
2
3
…
29 30