Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
0
votes
1 answer

Stream CSV data in Kafka-Python

Am sending the CSV data to Kafka topic using Kafka-Python. Data is sent and received by Consumer successfully. Now am trying to stream a csv file continuously, any new entry added to the file should be automatically sent to Kafka topic. Any…
Jim Macaulay
  • 4,709
  • 4
  • 28
  • 53
0
votes
1 answer

How to connect Kakfa Consumer from different server using Python

I am not able to receive any data from Kafka server to my consumer server. Kafka Server is hosted in 192.168.1.1 and consumer is hosted in 192.168.1.2 Using kafka-python lib. Below is my example from kafka import KafkaConsumer consumer =…
Harsh
  • 1
  • 1
0
votes
0 answers

How to fix error: kafka.errors.NoBrokersAvailable: NoBrokersAvailable?

I have setup docker-compose.yml as version: '2' services: zookeeper: image: confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 kafka: image: confluentinc/cp-kafka:latest …
Shubham Joshi
  • 56
  • 3
  • 11
0
votes
1 answer

How to solve kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs?

My docker-compose.yml code is: version: '2' services: zookeeper: image: confluentinc/cp-zookeeper:latest environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 kafka: image: confluentinc/cp-kafka:latest …
Shubham Joshi
  • 56
  • 3
  • 11
0
votes
1 answer

How to aggregate json data in kafka consumer using python?

my data in produced in KAFKA Transactions topic is like below: ConsumerRecord(topic='Transactions', partition=0, offset=3, timestamp=1591277946735, timestamp_type=0, key=None, value={'transaction_id': '9495601361', 'account_number': 14,…
Shubham Joshi
  • 56
  • 3
  • 11
0
votes
0 answers

I can't setup my enviroment to connect my consumer to docker kafka

I've tried to create a Python Consumer for kafka running on Docker. I've read about kafka listeners but still not getting the right setup. I'm new at Docker and also to kafka. Here's my docker-compose file: version: '2' services: zookeeper: …
Josué Molina
  • 1
  • 1
0
votes
2 answers

Find the topics under a consumer group in Kafka using Kafka-Python

Input to my python script is the name of a consumer group and the output is supposed to be the list of topics under the consumer group. My code currently, doesn't return the topics for which the current offset is -1. Is there a better way, by which…
user11322908
  • 23
  • 5
0
votes
1 answer

cant read\write from kafka topic

I have Kafka running on a container on my desktop. I can connect to it just fine using a tool called "Kafka tool" where I can see my topics for example. I'm having issues reading and writing to/from a Kafka topic. what's annoying me is that it…
Diego
  • 34,802
  • 21
  • 91
  • 134
0
votes
2 answers

kafka-python: how to check if broker list available in advance instead of showing error logs

My snippet looks like this, and the logic is if the brokers are available then the producer is not None. try: self.producer = KafkaProducer(bootstrap_servers=broker_list) except kafka.errors.NoBrokersAvailable: …
DennisLi
  • 3,915
  • 6
  • 30
  • 66
0
votes
1 answer

`for .. in KafkaConsumer`: how to survive `eof`?

for consumed_record in KafkaConsumer(config.kafka_topic, bootstrap_servers=config.kafka_bootstrap_servers, …
Zazaeil
  • 3,900
  • 2
  • 14
  • 31
0
votes
1 answer

Kafka Python Consumer Commit API

I need to do a programmatic commit using kafka-python api. From the the documentation i see that I need to use the api below, but couldn't find a good usage example: It needs offsets as a dictionary of TopicPartition and OffsetMetadata. I was…
hakish
  • 3,990
  • 7
  • 39
  • 56
0
votes
1 answer

Unable to set up Kafka and Python with docker

I am able to set up Kafka with docker but when I try to access it with python, I am unable to do that. I am able to do that if I install python inside kafka shell, but outside kafka shell and inside docker python, I am unable to use kafka. My…
Divyanshu
  • 11
  • 5
0
votes
0 answers

JSON error "Expecting value: line 2 column 11 (char 12)"

I am running the below code to fetch the data from an apache kafka consumer in python. from kafka import KafkaConsumer import json consumer = KafkaConsumer("DataEngFuelData", bootstrap_servers=['localhost:9092'], group_id=None, …
Vinayak Parab
  • 1
0
votes
0 answers

kafka-python timeout when producer send msg to topic in docker container

I have three kafka hosts which run in real hosts; I want to send msg to kafka in a docker container, This container can connect to kafka hosts; When I send msg in docker container use kafka-python,It always taken timout error,like this: File…
wsf1990
  • 195
  • 2
  • 12
0
votes
1 answer

Cannot connect to kafka docker container from outside client (wurstmeister images)

There are so many answers for this question that I ended up being totally confused about how I can connect to Kafka docker container from an outside client. I have created two docker machines, a manager and a worker with these…
e7lT2P
  • 1,635
  • 5
  • 31
  • 57
1 2 3
…
29 30