Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
0
votes
0 answers

Calling async function within Tornado overriden methods

I am trying to call async functions in tornado overriden methods. Initially, the function generate() is syncronous, but that seems to block the entire program, including on_connection_close method class Reader: def __init__(self, topic,…
jigiy43106
  • 21
  • 2
0
votes
0 answers

python-kafka asyncio is blocking

I am trying to create a websocket server that receives input from user, and at the same time read from kafka and continuously send data to user. I am having trouble getting both to run concurrently. async def sender(websocket): consumer =…
jigiy43106
  • 21
  • 2
0
votes
1 answer

Python Kafka read message once only

I am trying to read the earliest/oldest message which has not been read before in Kafka. import json from kafka import KafkaConsumer from kafka import KafkaProducer ORDER_KAFKA_TOPIC = "order_details" ORDER_CONFIRMED_KAFKA_TOPIC =…
jigiy43106
  • 21
  • 2
0
votes
0 answers

Unable to consume Eventhub messages using Kafka API

I am trying to read messages from Azure Event Hub using kafka API. But I am unable to connect to the event hub. The bootstrap_connected() function always return false and the code stops there. It is not going to print message part. Below is my…
potterson11
  • 147
  • 7
0
votes
0 answers

DNS lookup failed for kafka - ERROR kafka.conn:conn.py

I am getting the following error: WARNING kafka.conn:conn.py:1527 DNS lookup failed for kafka:9092, exception was [Errno -3] Temporary failure in name resolution. Is your advertised.listeners (called advertised.host.name before Kafka 9) correct and…
Angel
  • 25
  • 6
0
votes
0 answers

Kafka in GKE kubernetes internal communication between pods is not working

I'm trying to set up Kafka in Kubernetes (GKE) with pods for zookeeper, 1 broker, and 2 python apps: producer and consumer. However, I'm not able read and write to the topic correctly. I have set up zookeeper like this: apiVersion: v1 kind:…
Kalleni
  • 39
  • 1
  • 5
0
votes
0 answers

How to consume messages from Kafka consumer to Django app in new docker container?

The Django server is going stuck if Kafka-comsumer starts. What should I do? I want to run this consumer as a sidecar container. Can anyone help me? consumer = KafkaConsumer("my-topic", bootstrap_servers=\['{}:{}'.format(HOST,…
Abhay Braja
  • 26
  • 6
0
votes
0 answers

Connect kafka on cloudera cdh using kerborose authentication with keytab file

We have a keborosed cloudera cluster setup for apache kafka, I am trying to connect it using python and I have installed the MIT kerborose for windos system Main code: import os from kafka import KafkaConsumer import…
Vishal Kamlapure
  • 590
  • 4
  • 16
0
votes
1 answer

When creating a local Kafka instance for kafka-python, is it a good idea using Kafka 2.8+ that don't need Zookeeper?

In April 2021 Kafka released a version with early access to eliminating its dependency on Zookeeper. I've read many posts (mostly from 2021) saying that it was still not a good idea to use those versions on prod becasue they were too new. Every…
Vicky C
  • 11
0
votes
0 answers

Unable to connect to On Prem Kafka topic from GCP

I am trying to consume a kafka topic using SASL but getting an Error which has no documentation : ERROR:kafka.conn:Connect attempt to host=:9094 [IPv4 ('', 9094)]> returned…
Sam
  • 161
  • 1
  • 9
0
votes
1 answer

read kafka topics from python

This code doesn't display any messages successfully What's wrong? from kafka import KafkaConsumer consumer = KafkaConsumer( 'topictest', bootstrap_servers=['server:9092'], auto_offset_reset='None', enable_auto_commit=True, …
VLBigo
  • 9
  • 4
0
votes
0 answers

Consumer do not join a consumer group

I am using kafka-python and I want to consume messages from a topic. For monitoring reasons, I want to create a consumer and assign it to a consumer'group. I am using the following functions: server = KafkaConsumer(application.name,…
Andrea Fresa
  • 351
  • 2
  • 18
0
votes
1 answer

Kafka consumer failing due to nulls for value serializer

I have a Kafka consumer reading from a producer, but sometimes upon a record deletion from a table, null values get picked up by the consumer resulting in NoneType error message in the value serializer. How do I rewrite this 'value_serializer=lambda…
Shaggy
  • 159
  • 1
  • 1
  • 7
0
votes
1 answer

how to poll/listen to a Kafka topic continuously?

I have to listen to a topic on Kafka and consume the messages being published on it. My piece of code is working fine but sometimes gets stuck in the try block as poll returns empty dictionary. Please suggest a better way to poll the topic.…
WW_
  • 99
  • 10
0
votes
1 answer

Reason on uneven lag on kafka partitions

I have a kafka topic partitioned into 60 partitions. Producer distributes messages in a default round-robin manner. So far so good. But when it comes to consume, a lag is growing. Nothing surprising. The surprise for me is how the lag is sometimes…
Piotr Pradzynski
  • 4,190
  • 5
  • 23
  • 43
1 2 3
…
29 30