Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
0
votes
1 answer

Convert avro serialized messages into json using python consumer

from kafka import KafkaConsumer import json import io if __name__ == '__main__': # consumer = KafkaConsumer( # 'ldt_lm_mytable', # bootstrap_servers = 'localhost:9092', # auto_offset_reset = 'earliest', # group_id =…
Abhra Sarkar
  • 45
  • 7
0
votes
1 answer

Kafka-Python, Producer send record but Consumer don't receive it

I'm having trouble using kafka for my python code. I use python 2.7.5, and the package kafka-python. I want to send csv's (300000 rows, 20 fields per row) through a kafka topics. Before that I serialize each row into a json file and up to here,…
RxxxxSxxxx
  • 1
  • 1
0
votes
0 answers

Reading All the Even Offsets in a Kafka topic

I have this really strange ask and is unconventional in terms of Kafka usage. I would like to read all the even offsets in a Kafka topic. How do I do that using Kafka-Python package?
LifeAndHope
  • 674
  • 2
  • 10
  • 27
0
votes
1 answer

Implement Kafka Producer Consumer Chat with Queue

I have created implemented Kafka Producer-Consumer messaging with Topic using python.How can I do the same with Queue so that the message will be only devilered to a single consumer . This is my Producer code # Import KafkaProducer from Kafka…
user8506777
  • 43
  • 8
0
votes
1 answer

How do I get the data(messags) coming in the last 'n' mins on the consumer side for Kafka-python

I have a producer that sends messages in the form of a list of dictionaries. The consumer reads messages but I am not able to fetch the messages that arrive in the last 'n' minutes. Any help would be appreciated.
anonymous
  • 3
  • 3
0
votes
0 answers

Kafka to postgres without kafka connectors

I am trying to downstream data from kafka topics to a relational db like postgres. I don't want to use kafka connect or jdbc connectors. Is their any way I can use raw queries to export data from kafka to postgres. I will be using python to…
androidbear
  • 9
  • 1
  • 4
0
votes
0 answers

How to check if topic Kafka is purged

I want to check if my topic is empty after changing a retention.ms to 100ms. I try to consume from my topic like this : consumer_kafka = KafkaConsumer(topic, bootstrap_servers=bootstrap_servers, auto_offset_reset='latest', …
LilyAZ
  • 133
  • 3
  • 10
0
votes
0 answers

Can receive messages using cli, but not getting messages from kafka python

I am using AWS MSK Kafka managed streaming service and all of a sudden I am not receiving any messages through my consumer. I am using kafka-python and the weird thing is that when I open up my cli and create a new consumer with the exact same code,…
joethemow
  • 1,641
  • 4
  • 24
  • 39
0
votes
1 answer

Underlying kafka version in kafka-python

I have a topic from which I need to consume and process data and I am using kafka-python package to do so. I am facing a lot of issues related to rebalancing, slow data consumption across a few partitions, and want to confirm if it because of any…
kowshik tallam
  • 13
  • 3
0
votes
0 answers

Not able to produce and consume kafka message from my local machine to ubuntu VM

I am trying to produce kafka message from my local machine to Ubuntu vm. Telnet commad says successfully connected to host. PS C:\Users\harshal> Telnet harshal-VirtualBox 9092 Successfully connected to Host: "harshal-VirtualBox" on Port:…
HNM
  • 1
  • 3
0
votes
2 answers

Create a Kafka topic if it does not already exist with Python-kafka admin client?

With Python-kafka admin client I know how to create and delete topics. But is there any way to check if that topic exists or not? If not then only I wanted to create it. How can I do that in python -kafka admin client? I didnt see anything in the…
1689
  • 117
  • 1
  • 10
0
votes
1 answer

ERROR Unknown error when running consumer: org.apache.kafka.common.errors.SerializationException: Unknown magic byte

Am trying to send messages from python to kafka consumer. But am getting error as ERROR Unknown error when running consumer: org.apache.kafka.common.errors.SerializationException: Unknown magic byte! Python is properly getting the data from twitter…
Jim Macaulay
  • 4,709
  • 4
  • 28
  • 53
0
votes
1 answer

How to debug AvroConsumer in confluent kafka?

I'm trying to read Kafka from python but recieve message is None , No errors in CLI. I'm using port forwarding to destination host via putty, and than test ports over telnet - it work's fine. Moreover, i'm using kafkacat on Debian (WSL) and it…
shabelski89
  • 19
  • 6
0
votes
0 answers

Kafka specific Consumer group is rebalancing

I'm using kafka on docker (note : with wurstmeister image). I have three consumers groups (each group contains 3 consumers). Two groups are running without problems, the last one added is running bad (many rebalancing on kafka logs) : consumer code…
Chris
  • 234
  • 3
  • 16
0
votes
1 answer

How to combine output from Kafka Topics

Following is my use case scenario where one application will push data to three different kafka topics (there is unique app id) and the output will go to the subsequent queue 4 and queue 5. I have already implemented the pipeline shown below. The…
Kanchan Sarkar
  • 423
  • 2
  • 11
1 2 3
…
29 30