Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
0
votes
0 answers

Kafka: Using multiple consumers to reduce consumer lag doesn't work

hi I have a Kafka cluster using 3 Brokers (EC2 instances) and there is a topic with 100 partitions. I wanted to observe how much a messages per second a certain number of consumers can handle. So I've sent messages by [20, 100, 500, 1000, 5000,…
lee
  • 1
0
votes
0 answers

KafkaConsumer continues to have "CommitFailedError" after suggested fixes

Recently started working with Kafka in an offline environment. Tools used are: kafka-python (2.0.2), kafka_2.12-2.8.1 and zookeeper 3.7.0, running everything on LAN network of machines using Windows 10. I can't share the code, because offline, but…
Robert Testerman
  • 11
  • 6
0
votes
1 answer

kafka-python: produce and consume messages from same topic at the same time by running concurrent process/scripts

Kafka set up locally: bin/zookeeper-server-start.sh config/zookeeper.properties bin/kafka-server-start.sh config/server.properties and example test topic to store data is created: bin/kafka-topics.sh --create --topic fortest --bootstrap-server…
Назарій Кушнір
  • 77
  • 10
0
votes
0 answers

How to handle expired producer batches

I made a Kafka producer using kafka-python to send records to a remote broker. If I have a problem of network connection during more than request_timeout_ms (here 20s), the callback sends me this exception : KafkaTimeoutError: Batch for…
noam
  • 1
  • 2
0
votes
1 answer

TypeError: partitions must be TopicPartition namedtuples

I want to use KafkaConsumer from kafka-python to consumer the first N messages in a topic: from kafka import KafkaConsumer as kc import json bootstrap_servers = ['xx.xxx.xx.xxx:9092'] topic_name = 'my_topic_name' consumer = kc(topic_name,…
Tristan Tran
  • 1,351
  • 1
  • 10
  • 36
0
votes
1 answer

How can I commit a message offset to Kafka topic after consumption?

It seems that the offset is not being committed. I am using kafka Python package. This is my code from kafka import KafkaConsumer consumer = KafkaConsumer( 'quickstart-events', bootstrap_servers=['localhost:9092'], …
jebaseelan ravi
  • 185
  • 2
  • 10
0
votes
2 answers

Log offset, partition and topic from Kafka message

I want to log or print offset, partition, and topic from Kafka's message. I can print the message value but I want to see which offset and partition from Kafka using python so that I can debug my code from kafka import KafkaConsumer consumer =…
Rahul
  • 87
  • 1
  • 1
  • 12
0
votes
0 answers

Fetching data of kafka queue within a certain defined time range

I am trying to extract message of kafka queue based on certain time ranges, say between Oct 2, 1 PM and Oct 2, 3 PM. How can I explicitly define this time range and just pull data off these time ranges. bootstrap_servers =…
Scott Nash
  • 1
0
votes
1 answer

Create a new kafka topic using python from a container connecting to another container

I would like to create a new kafka topic using python, and I get an error when I try to create a KafkaAdminClient using server="kafka:9092": self._kafka_admin = KafkaAdminClient( bootstrap_servers=[server], …
Tavis
  • 71
  • 6
0
votes
1 answer

How can i consume kafka data which is produced in python

I am getting confused in how to create a kafka producer and consumer in pycharm. i have created a produce.py from time import sleep from json import dumps from kafka import KafkaProducer producer = KafkaProducer( value_serializer = lambda…
zedu3
  • 1
  • 1
0
votes
1 answer

kafka.errors.NoBrokersAvailable: NoBrokersAvailable

I'm trying to use kafka-python for accessing Kafka in a Docker container. The dockerized app from which I'm trying to connect to Kafka is in another container in the same network. The error appears when I try to initialize a KafkaAdminClient…
Tavis
  • 71
  • 6
0
votes
1 answer

Python script is running in the IDE, but not in the terminal (Kafka)

This may or may not be Kafka related, but I encountered this while learning Kafka. I've got a python producer script that looks like this: from kafka import KafkaProducer from json import dumps class Producer: def __init__(self): …
John Kealy
  • 1,503
  • 1
  • 13
  • 32
0
votes
0 answers

Protobuf message decoding error using kafka-python ParseFromString

I have a protobuf model like below: i.e. protoObj sample { barcode: "BAR000002" requisition_id: "ORDER-1" patient_id: "P00002" cancer_type: "CRC" } assay { key: "lunar2" title: "Lunar2" version: "3.0" } lunar2 { type:…
suyashnatural
  • 1
  • 1
0
votes
0 answers

How to check the active number of consumers in a consumer group using kafka-python?

I have a consumer group with 15 consumers in it. When I start the process, around 200 records are processed by these 15 consumers in one minute, but later on as the number of incoming records goes down, I believe consumers start losing connection…
Sachin Dev Sharma
  • 196
  • 2
  • 11
0
votes
1 answer

Schema for json message in aiokafka

How can I add a schema for json message in aiokafka? Kafka Connect cannot work without it. import asyncio import json import random import aiokafka from faker import Faker def serializer(value): return json.dumps(value).encode() async def…
Альберт Александров
  • 807
  • 1
  • 15
  • 27
1 2 3
…
29 30