Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
5
votes
1 answer

How to implement request-reply (synchronous) messaging paradigm in Kafka?

I am going to use Kafka as a message broker in my application. This application is written entirely using Python. For a part of this application (Login and Authentication), I need to implement a request-reply messaging system. In other words, the…
Arashsyh
  • 609
  • 1
  • 10
  • 16
5
votes
1 answer

How to produce Kafka messages with JSON format in Python

How to delete quotes and send data like on original format The original JSON-format is: { "@timestamp": "2020-06-02T09:38:03.183186Z" } This data in another topic "{\"@timestamp\": \"2020-05-25T17:40:47.582778Z\"}" This is a code of sending data…
LOTR
  • 113
  • 1
  • 1
  • 10
5
votes
0 answers

confluent-kafka-python: how to timeout initial connection when Broker is not available?

I am using confluent-kafka-python and see that it hangs infinitely when I try to connect to a broker which is down. I can't seem to apply any time out settings I found in the docs: from confluent_kafka import Consumer conf = {'bootstrap.servers':…
Artem Trunov
  • 1,340
  • 9
  • 16
5
votes
0 answers

How to handle connection issues with kafka using the python kafka library?

I have a serverless function that's trying to send some data to kafka. Sometimes it works and sometimes the connection just drop and the data is lost. The reason for this is that the library for kafka is not raising the exception but it's adding…
Yuv Joodhisty
  • 71
  • 1
  • 4
5
votes
1 answer

Unable to send messages to kafka from django application using kafka-python due to KafkaTimeoutError

I have a Django based web application in which I am trying to integrate Kafka with the help of this library named kafka-python. However when I am trying to send a message to a particular topic I am getting a time out error stating : Traceback (most…
Paras
  • 3,191
  • 6
  • 41
  • 77
5
votes
1 answer

What is the correct way to manually commit offset to kafka topic

I have a consumer script which processes each message and commits offsets manually to the topic. CONSUMER = KafkaConsumer( KAFKA_TOPIC, bootstrap_servers=[KAFKA_SERVER], auto_offset_reset="earliest", max_poll_records=100, …
Shakeel
  • 1,869
  • 15
  • 23
5
votes
2 answers

KafkaTimeoutError('Failed to update metadata after 60.0 secs.')

I am writing a Kafka producer using Python 3.6,the Python-kafka client version is 1.4.4。The Kafka version is: 2.1.0 & 1.1.1(two version are tried),but when I write a message to producer,throw this error: KafkaTimeoutError('Failed to update metadata…
Dolphin
  • 29,069
  • 61
  • 260
  • 539
5
votes
4 answers

How to create Kafka-python producer with ssl configuration

I'm trying to create kafka producer with ssl. I need information on how to set SSL parameters in the constructor, the information provided in kafka-python client is not descriptive enough. What are the ssl_certfile, ssl_cafile, ssl_keyfile…
Kalyanapu RajaShekar
  • 71
  • 1
  • 1
  • 5
5
votes
1 answer

Kafka Consumer poll messages with python

I have problems with polling messages from Kafka in a Consumer Group. My Consumer Object assigns to a given partition with self.ps = TopicPartition(topic, partition ) and after that the consumer assigns to that…
soa
  • 263
  • 2
  • 5
  • 17
5
votes
0 answers

Python `Error in atexit._run_exitfuncs`

I am trying to use kafka MultiProcessConsumer but I am getting following error. It seems like error is related to multithreading in python Here is the code which I am using. simple.py from kafka import SimpleProducer, SimpleClient, SimpleConsumer,…
Avinash
  • 2,093
  • 4
  • 28
  • 41
4
votes
2 answers

Python-Kafka: Keep polling topic infinitely

I am using python-kafka to listen to a kafka topic and use that the records. I want to keep it polling infinitely without any exit. This is my code below: def test(): consumer = KafkaConsumer('abc', 'localhost:9092',…
mifol68042
  • 183
  • 1
  • 4
  • 18
4
votes
0 answers

Faust Table Lookup Performance

I am trying to perform Lookup for messages arriving in a Kafka topic (say "datatopic"). Lookup source is another Kafka Topic (say "lookuptopic"). To do this using faust, I created a Table using lookuptopic and created an agent to update this table…
some_user
  • 315
  • 2
  • 14
4
votes
2 answers

What is the best way of polling at regular intervals a Kafka Consumer while using kafka-python?

I have multiple Producers which are feeding data into Kafka. I wish to run a Consumer every hour to get all the accumulated data in at once and process it further. The options which I thought of are: Use a python thread and use an equivalent of…
Varun Shridhar
  • 143
  • 1
  • 1
  • 8
4
votes
4 answers

How to describe a topic using kafka client in Python

I'm beginner to kafka client in python, i need some help to describe the topics using the client. I was able to list all my kafka topics using the following code:- consumer = kafka.KafkaConsumer(group_id='test',…
Ashwin
  • 439
  • 1
  • 7
  • 23
4
votes
2 answers

Reset kafka LAG (change offset) within consumer group in Kafka-python

I found this where I reset my LAG with the kafka-consumer-groups.sh tool How to change start offset for topic? but I am needing to reset it within the application. I found this example, but it doesn't seem to reset it. kafka-python read from last…
Kevin Vasko
  • 1,561
  • 3
  • 22
  • 45
1 2 3
…
29 30