Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
0
votes
2 answers

dockerized Kafka service stuck at producer.send

I am very new to docker and Kafka, and have a simple kafka python publisher shown below The following are in my dockerfile: FROM python:3.10 WORKDIR /app COPY . /app RUN pip install --user pip==23.0.1 && pip install pipenv && pipenv install…
Alejandro
  • 879
  • 11
  • 27
0
votes
2 answers

Python How to pass continuous generated data stream between methods

I have a method that continuously generates data and prints them on the console. Let's say something simple like generating random number: def number_generator(): while True: random.randint(1,100) time.sleep(0.5) return I…
Varun Pius Rodrigues
  • 337
  • 2
  • 8
0
votes
1 answer

Can produce message with kafka-python, but can't with confluent-kafka

I try to use confluent-kafka, but can't manage to send message to topic. I did the same with kafka-python, and message sends. What i am missing? Thank you! i use this code # using confluent-kafka from confluent_kafka import Producer producer =…
Vitalik
  • 1
  • 3
0
votes
1 answer

Python - Optimal way to check for a condition with a timeout?

Wrote two functions, 1. uploads a gzip file to Artifactory and 2. listens to the Kafka topic. As further steps, wanted to validate whether the upload is successful within 5 minutes by listening to the Kafka topic for the keyword - push_status: True,…
Goku
  • 482
  • 1
  • 7
  • 22
0
votes
0 answers

Getting 403 Error while trying to stream data to kafka from tweepy

from tweepy.streaming import StreamListener import tweepy from tweepy import OAuthHandler from tweepy import Stream from kafka import SimpleProducer, KafkaClient import json access_token = "" access_token_secret = "" consumer_key = …
Len Kagamine
  • 13
  • 3
0
votes
0 answers

Cannot connect to Kafka broker, BrokerConnection error 111

I have a container in the same docker-compose that is trying to connect to my broker. It suddenly stopped connecting and is resulting with error: 2023-02-22 14:24:00 ERROR:kafka.conn:Connect attempt to
LauraAlice
  • 33
  • 6
0
votes
2 answers

When kafka consumer connected to a broker by domain name, if the ip changed after broker crash, will the kafka consumer reconnect?

My kafka consumer is connected to a cluster of brokers. There is a domain name server in between. Each broker has a domain name associated with an IP address. Problem is, the IPs are not static and for some reason, I have to restart the broker. The…
sovon
  • 877
  • 2
  • 12
  • 28
0
votes
1 answer

Kafka Consumer Committing With Auto-Commit Disabled

I'm missing events when reading from a Kafaka queue because the consumer is updating the offset without an explicit commit even when enable_auto_commit is disabled. from kafka import KafkaClient, TopicPartition topic_name = "my_topic" consumer =…
Gree Tree Python
  • 529
  • 1
  • 6
  • 22
0
votes
0 answers

Alter client quotas using kafka-python

I'm using kafka-python library to manage my kafka cluster. I want to add the quotas features (alter/describe), these features are not yet implented in kafka-python library but they are available in kafka protocol doc here. AlterClientQuotas API…
a.anghour
  • 21
  • 4
0
votes
1 answer

Is there a suitable python library for doing stream processing with Kafka topics?

I am trying to find a suitable Python library to do stream processing with streams Kafka topics, Kafka streams. Specifically, I am looking for libraries that support the following operations. KStream-to-Kstream KTable-to-Ktable KTable-to-KTable…
Eagle1992
  • 3
  • 1
0
votes
1 answer

kafka produce messages in consumer

I have an application with the need to pass messages into multiple layers of processing. I need to do this because all the new messages should be put into the first generic topic so they can be processed to calculate a type and after that, they…
Firouziam
  • 777
  • 1
  • 9
  • 31
0
votes
1 answer

Kafka producer and consumer not working properly on python with docker

Im working on a project that uses kafka producer and consumer in order to acquire articles (with specific topics) from a news_api every two hours and then with a consumer save them in a mongodb. So i made three classes one for KafkaAdminClient one…
chris
  • 1
  • 1
0
votes
1 answer

enable_auto_commit=False, but still offsets are committed automatically

I'm using kafka-python==2.0.2, and have disabled auto_commit but still if I don't commit through code, offsets are automatically getting committed In the below code even if I comment out self.consumer.commit_async(callback= ...., offsets are still…
Swastik
  • 144
  • 2
  • 19
0
votes
1 answer

kafka broker not collect data - CentOS7

I made a simple configuration with Kafka. I ran it as a service on 2 Centos servers. While I can send and see logs to the logs topic with my python scripts that I created on one of my servers, I only send them on my other server and I cannot view…
Buket
  • 1
0
votes
1 answer

How to perform async commit when using kafka-python

I'm using kafka-python library for my fastapi consumer app and I'm consuming messages in batch with maximum of 100 records. Since the topic has huge traffic and have only one partition, consuming, processing and committing should be as quick as…
Swastik
  • 144
  • 2
  • 19
1 2 3
…
29 30