Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
0
votes
1 answer

Python client for Managing Kafka ACL

Do we have any python client for managing Kafka ACL? I know we have Java Admin client and same can be performed using Kafka scripts (kafka-acl.sh) but cannot find one for python client. confluent python library doesn't support it. Came across…
Vikash Mishra
  • 123
  • 2
  • 10
0
votes
2 answers

Why do I get Intermittent "NoBrokersAvailable" errors, with python application (kafka-python Producer)?

I’m trying to make a connection to Kafka on AWS. Communication is done over TLS.
Kafka-python==2.0.2 is used, and the KafkaProducer configuration is as follows: From Kafka import KafkaProducer producer = KafkaProducer( …
bowoojang
  • 3
  • 2
0
votes
1 answer

Error "certificate has expired" when connecting to Confluent Cloud Kafka

I created a Kafka cluster at https://confluent.cloud/. I am trying to write a simple producer, with kafka-python. Here is my producer code: from kafka import KafkaProducer producer = KafkaProducer( …
Wallace Kelly
  • 15,565
  • 8
  • 50
  • 71
0
votes
1 answer

Set consumer offset

If I want get all messages from to start offset, I run this shell command: /usr/share/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic 1-codicefiscale-21032022122736-i --partition 0 --offset 5 How can I do the same…
user18480960
0
votes
0 answers

Kafka python consumer not receiving new data inserted into the source data

I have my source database as Mysql and sink database as Cassandra. Python producer code establishes connection with Mysql and sends data to kafka topic, in my case the topic name is demo. The consumer code then data to Cassandra. Here is my producer…
Tej Patel
  • 3
  • 4
0
votes
1 answer

list_consumer_group_offsets is not returning all the partitions and their offsets

I am using kafka-python to get the last offset being consumed by a consumer: def get_consumer_group_offset(group): topic = '' g = create_admin() returnlist = [] counter = 0 end_offsets =…
Atheer Abdullatif
  • 182
  • 1
  • 15
0
votes
0 answers

Mocking KafkaConsumer with pytest

Recently I've been trying to learn unit testing and now I'm on the concept mocking. I'm writing my code with python and using pytest library to test my code. I've a simple factory class has only one function that creates a KafkaConsumer, and I want…
Oğuz Öztınaz
  • 165
  • 9
0
votes
1 answer

How to consume a specific maximum number of messages from a Kafka Topic?

I wish to consume a specific number of maximum messages per consume from a kafka topic. I have used the following code: consumer = KafkaConsumer(topic, bootstrap_servers=[server], max_poll_records=2, api_version=(0, 10)) for message in consumer: …
Suganth
  • 35
  • 1
  • 7
0
votes
2 answers

Python Kafka - max messages in file

Is there any way to get just one message, save it to a file, and terminate the script? I'd like to loop this over so that it saves me each message to a separate file. From the cmd level, I had a parameter: max-messages 1 but in python I haven't…
danio900409
  • 265
  • 2
  • 6
  • 22
0
votes
2 answers

How to consume messages from kafka producer in batches (kafka-python)

I am having a kafka producer and consumer in python. I wish to consume messages from kafka producer in batches, let's say 2. From the producer, I have been sending email data like the following: [{ "email" : "sukhi215c@gmail.com", "subject":…
Suganth
  • 35
  • 1
  • 7
0
votes
1 answer

KafkaProducer does not send messages in multiprocessing.Process

I got the following concept, which is a simplified example of a real app. The problem is that the producer will not produce messages while working in a multiprocessing.Process process. import logging import multiprocessing import os from kafka…
Ivan
  • 316
  • 3
  • 15
0
votes
1 answer

Create duplicate topic in kafka using python

Is there a way we can create a copy of the kafka topic using kafka-python package? I am trying to create a backup of a topic before performing few operations on the data.
Anish
  • 114
  • 1
  • 2
  • 10
0
votes
1 answer

Topic is not available during auto-create initialization

I'm trying to get a kafka consumer to subscribe to 2 topics, yet whenever I try and assign more than one Topic I get this…
LiorFurman
  • 1
  • 1
0
votes
0 answers

Can I use poll and iterate in kafka-python together?

I am working with kafka-python. I would like to fetch big amount data from kafka topic till its empty. Then do some action. Then start consuming online new events from that topic. My kafka-python implementation: consumer = KafkaConsumer( …
user3691223
  • 353
  • 2
  • 11
0
votes
0 answers

kafka-python KafkaConsumer: read latest message non-blocking

How can I grab the latest produced message from a Kafka Producer without looping? It has to be quick, so I cannot afford to wait for a new message to come in and instead wish to read the most recent message. What I wish to do is equivalent to the…
Mason McGough
  • 532
  • 5
  • 13
1 2 3
…
29 30