Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
3
votes
1 answer

How to programmatically create topics using kafka-python?

I am getting started with Kafka and fairly new to Python. I am using this library named kafka-python to communicate with my Kafka broker. Now I need to dynamically create a topic from my code, from the docs what I see is I can call create_topics()…
Paras
  • 3,191
  • 6
  • 41
  • 77
3
votes
4 answers

reading only specific messages from kafka topic

Scenario: I am writing data JSON object data into kafka topic while reading I want to read an only specific set of messages based on the value present in the message. I am using kafka-python library. sample messages: {flow_status: "completed",…
Prabhanj
  • 262
  • 2
  • 3
  • 16
3
votes
1 answer

Getting latest message timestamp in my Kafka queue

I have Kafka 0.10.0, which, if I understand correctly, adds timestamps to all of the messages. For monitoring purposes, I want to pull out the timestamp of the newest message for a given topic. I did not see an API field for it in any of the Python…
Konstantin Tarashchanskiy
  • 2,762
  • 2
  • 31
  • 46
3
votes
1 answer

How does a sawtooth pattern of Kafka consumer lag emerge?

Some of my Kafka consumers (but not all) show an interesting pattern regarding their lag. The following image shows two good examples: dark-blue: about 200 messages per second in topic 32 partitions 1 consumer in group (Python client, running on…
Tobias Hermann
  • 9,936
  • 6
  • 61
  • 134
3
votes
2 answers

Not able to send large messages to Kafka

I want to send a large message from producer to Kafka so I've changed below properties. Broker (server.properties) replica.fetch.max.bytes=317344026 message.max.bytes=317344026 max.message.bytes=317344026 max.request.size=317344026 Producer…
Vatsal Jagani
  • 83
  • 1
  • 8
3
votes
0 answers

Partition are not getting assigned to Kafka consumer instance

When I start consumer with single instance it is show in consumer group but it is not consuming data from topic. After that if I start another consumer and my first consumer start consuming data but latest consumer instance doesn't have any…
user1534977
  • 31
  • 1
  • 4
3
votes
0 answers

kafka-python MessageSizeTooLargeError, even after Increasing max_request_size

I am facing an issue while sending a huge json (~40MB) from Kafa producer to consumer. I referred other similar posts on StackOverflow but still unable to succeed. I set these parameters in my Producer and Consumer: producer =…
Aniruddh Khera
  • 111
  • 1
  • 13
3
votes
2 answers

Retrieving consumer group offsets in kafka using python library

I have python script and I need to retrieve the current consumer group offset for a set of consumers reading from a kafka topic, using a kafka1 broker cluster. These are native kafka consumers that store the offset in the kafka cluster, not in…
andyJ
  • 61
  • 1
  • 4
3
votes
2 answers

Kafka partition Lag increasing

I have an application which uses Kafka 1.0 as a queue. The Kafka topic has 80 partitions and 80 consumers running. (Kafka-python consumers). By running the command : ./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group mygroup …
ashdnik
  • 648
  • 3
  • 8
  • 20
3
votes
2 answers

Messages are not delivered with kafka python

I'm trying to set up a simple Kafka application with kafka-python. I've been trying to get some of the examples I found online to work but can't seem to make it. I have a kafka instance running in a docker container. I tested the shell tools and the…
Aleksandar Savkov
  • 2,894
  • 3
  • 24
  • 30
3
votes
1 answer

Kafka producer not picking new partitions

I'm new to Kafka, and am trying to build a service to service messaging platofrm on it. Here's my setup: Kafka 0.9.0.1Zookeeper 3.4.8kafka-python 1.3.3 My application creates a KafkaProducer from which I send a stream of messages to a single topic…
Saksham Gupta
  • 93
  • 1
  • 8
3
votes
4 answers

Using confluent-kafka python client in alpine container

I'm trying to run a simple python app that communicates with kafka. I'm looking to use an alpine container for it. Here's my current dockerfile (it's not optimal... just trying to get things working for now). FROM python:3.6-alpine MAINTAINER Ashic…
ashic
  • 6,367
  • 5
  • 33
  • 54
3
votes
3 answers

KafkaException: Wrong request type 18

I recently attempted to consume messages from a Kafka topic and was greeted with the following error message: [2016-08-15 17:56:05,025] INFO Closing socket connection to /192.168.34.11. (kafka.network.Processor) [2016-08-15 17:56:05,050] ERROR…
m81
  • 2,217
  • 5
  • 31
  • 47
3
votes
0 answers

Kafka Consumer Hangs on Startup

Due to one reason or another, we recently re-wrote our consumers and producers using different libraries than initially written in. However, I've been having some issues after the switch. Using Kafka 0.9.0.2: 8 partitions We consume from one…
user2402831
  • 693
  • 10
  • 22
3
votes
3 answers

Does Apache Kafka Store the messages internally in HDFS or Some other File system

We have a project requirement of testing the data at Kafka Layer. So JSON files are moving into hadoop area and kafka is reading the live data in hadoop(Raw Json File). Now I have to test whether the data sent from the other system and read by kafka…
johny
  • 61
  • 1
  • 3
1 2 3
…
29 30