Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
0
votes
1 answer

Why Kafka care about the hostname?

I did a test with code below to send data to topic. The kafka is kafka_2.12-1.1.0 The code are import kafka print(kafka.version.__version__) from kafka import KafkaProducer producer = KafkaProducer( …
Kramer Li
  • 2,284
  • 5
  • 27
  • 55
0
votes
0 answers

kafka consumer takes very, very long time to start

Defaults must be bad or something is very wrong. I run produce and consumer in separate REPL. import datetime import uuid import time import json import msgpack from kafka import KafkaProducer, KafkaConsumer from kafka.errors import…
mathtick
  • 6,487
  • 13
  • 56
  • 101
0
votes
1 answer

Sending a JSON file with multiple JSON objects through KAFKA

I have a file which contains multiple json documents in the following format. {"attribute1": "value1", "attribute2": "value2", "attribute3": "value3", "attribute4": "value4"} {"attribute1": "value11", "attribute2": "value12", "attribute3":…
jsn
  • 17
  • 3
0
votes
0 answers

How can I get consumer group list that is associated with topic in kafka-python?

I want to get a Kafka consumer group list by topic in kafka-python but I couldn't. How can I get consumer group list that is associated with topic in python code: from kafka import BrokerConnection from kafka.protocol.admin import * import…
Andrews Ashwin
  • 1
  • 1
0
votes
1 answer

python KafkaConsumer and Producer at the same time

In a python program I would like to write some messages to Kafka, then read the response with the same number of messages from a remote app on a different topic. The problem is that by the time I am done with sending messages, the other end already…
Artem Trunov
  • 1,340
  • 9
  • 16
0
votes
1 answer

How to make a consumer leave and enter a consumer group in kafka

So, I have a consumer group and I have a few distinct nodes, each acting as a consumer. Each node is supposed to perform some computation intensive task. I want to make a consumer join to this consumer group only when it has available CPU resources.…
Prashantha
  • 155
  • 1
  • 6
0
votes
0 answers

SASL_SSL using GSSAPI not able to connect

While I am trying to connect to kafka from other server using kafka-python. I am getting SSL certficate error. ssl_certfile='~/cacerts.pem',ssl_password='~~~~',sasl_mechanism='GSSAPI',sasl_kerberos_service_name='ssskafka' ) Traceback (most recent…
sbtota
  • 3
  • 2
0
votes
0 answers

Can't find module kafka

I have a python project that employees Kafka Queues, the project runs as expected but fails to run when run as part of a docker container.It keeps throwing this error can't find the module kafka This is how my Dockerfile.. FROM ubuntu:18.04 RUN …
Moses
  • 635
  • 8
  • 22
0
votes
1 answer

Listening to kafak on kubernetes from outside the cluster

I'm running a kubernetes cluster on google cloud platform and installed kafka (https://hub.kubeapps.com/charts/bitnami/kafka) on it using Helm charts. I also have another deployement that runs a python pod. I've exposed kafka and zookeper using…
rasthiya
  • 650
  • 1
  • 6
  • 20
0
votes
0 answers

consumer consuming the same message twice at the starting only

At the very first consuming, my consumer is consuming the same message twice, this only happens at the first consuming, after that it only consumes once. Attaching the consumer conf code below. please check for the corrections def…
Ankit Kumar
  • 41
  • 1
  • 7
0
votes
1 answer

Use AvroConsumer to get messages in partitions and topics

I want to be able to read messages that come in specific partitions of a topic and also messages in another topic like a I would with a simple Consumer. self.consumer = AvroConsumer(conf) parts = [TopicPartition('p_topic', 13), …
ealiaj
  • 1,525
  • 1
  • 15
  • 25
0
votes
1 answer

Kafka Python, How to track consumer started in a different process

I am fairly new to Python and just getting started with Kafka, so pardon my terminologies if I'm wrong somewhere. So I have a Django based web application, where I am sending a json messages through Kafka Producer within the same process. However…
Paras
  • 3,191
  • 6
  • 41
  • 77
0
votes
1 answer

kafka-python publish to all consumers

Is there a way to publish a message to all the partitions? I am using kafka-python==1.4.3. Understood from the documentation that key (optional) – a key to associate with the message. Can be used to determine which partition to send the message…
Yogeswaran
  • 357
  • 1
  • 13
0
votes
2 answers

Can a consumer read records from a partition that stores data of particular key value?

Instead of creating many topics I'm creating a partition for each consumer and store data using a key. So is there a way to make a consumer in a consumer group read from partition that stores data of a specific key. If so can you suggest how it can…
user119100
  • 31
  • 1
0
votes
1 answer

How to fetch ConsumerRecord from KafkaConsumer.poll() in python

I used kafka-python to process messages in a kafka cluster: consumer = KafkaConsumer('session', auto_offset_reset='earliest'] while True: dict = consumer.poll(500) for d in dict: print d.topic, d.partition, d.value That will give err…
richardj
  • 11
  • 1
  • 3
1 2 3
…
29 30