Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-python]

Confluent Kafka Python is a performant implemention of Kafka producers, consumers and the admin client in Python and it is based on librdkafka.

Confluent Kafka Python is a performant implemention of a Kafka client library based on librdkafka. You can use it to connect to Kafka brokers in Python.

219 questions
1
vote
1 answer

manual offset commit do not work as expected

I'm running confluent_kafka with 2.0.2 and test the manual offset commit code. I set up a confluentinc/cp-kafka:7.3.0 with a test-topic and 1 partition. I wrote a test script to test manual offset commmit with 'enable.auto.offset.store': False and…
WindMGC
  • 13
  • 3
1
vote
1 answer

Consuming messages from Kafka with different group IDs using confluent_kafka

I am using python and confluent_kafka I am building a Queue Management for Kafka where we can view the pending(uncommitted) messages of each topic, delete topic and purge topic. I am facing the following problems. I am using same group ID for all…
Ibrahim Khan
  • 184
  • 10
1
vote
0 answers

Confluent Kafka: Should I poll after producing a message when I have set the acks to 0?

I want to optimize my kafka producer (working in python using the confluent-kafka library). I have a topic for which acknowledgment of the leader receiving a message is not important. Therefore, I've set the acks to 0. I'm wondering something about…
Jeroen
  • 25
  • 1
  • 6
1
vote
0 answers

Unable to connect to Confluent Kafka in Python

I am trying to connect to confluent cloud hosted in Azure and consume the messages but on running the python code it give me the below error which I am unable to gauge. 2022-12-12 11:37:16,249 ERROR FAIL [python-example-consumer#consumer-1]…
potterson11
  • 147
  • 7
1
vote
2 answers

Unable to upgrade Python 3.11 due to error in Locust-plugin installation (Confluent-Kafka)

After upgrading Python to 3.11, Locust-plugin fails to install as one of its dependency throws error First it asked me to install VC++ > 14.0 After installing VC++ build tools 15, still it throws a different error Note: We do not see this issue…
Naga
  • 77
  • 1
  • 6
1
vote
1 answer

Run kafka consumer without while loop using python

I am using Confluentinc Kafka with Python & multi-threading. In this I have N worker threads running in parallel, whenever a thread completes its work it poll the message from kafka on demand. This whole job is done using the while loop. By using…
Ashrita Gupta
  • 11
  • 3
1
vote
0 answers

Unable to use logging library for kafka logs

I'm relatively new to confluent-kafka python and I am having an hard time figuring out how to send the logs from the kafka library to the program logger. I am using a SerializingProducer and, according to the docs here:…
fedmag
  • 55
  • 7
1
vote
1 answer

import error confluent_kafka mach-o file, but is an incompatible architecture (have (arm64), need (x86_64)))

I am using Macbook with M series chip. While running python which has confluent-kafka dependency, getting this error. Installed dependencies using pip3. Also installed librdkafka ImportError:…
wenky
  • 471
  • 1
  • 6
  • 14
1
vote
0 answers

Docker kafka.errors.NoBrokersAvailable: NoBrokersAvailable

I have dockerized my project that using kafka for communication, I am using python on consume part of my project, I used two different kafka image because of the NoBrokersAvaible error and I tried all of the combinations I guess. I want to consume…
Berkay Adyemir
  • 11
  • 1
1
vote
1 answer

consuming Kafka Avro massages in Python

I am trying to consume messages from Kafka Avro in Python. We have it in Java, and it's working, but when trying to consume it in the Jupyter notebook, Parsing does not work. I followed the example given by the documentation: (I've removed conf…
Armin ny
  • 25
  • 5
1
vote
1 answer

How can I initate retries on producer.poll() from the confluent_kafka library?

I'm very new to using kafka, not sure if I understand it yet tbh. However, I'm tasked with making sure it will retry if there is an error of any kind. I've done this is with an API if I recieve a response other than 200 for example, but I'm stumpped…
YourMomsDataEngineer
  • 59
  • 1
  • 5
1
vote
1 answer

How to Connect to kafka from python using truststore file?

I have properties.config file and kafka.client.truststore.jsk and I can easily get data from kafka with kafka-console-consumer in console using : kafka-console-consumer --topic test-topic --group group-id --bootstrap-server server:port…
Mohammad Anvari
  • 567
  • 5
  • 8
1
vote
0 answers

Python Kafka client with client_rack attribute

I have a requirement to set the client.rack attribute. I am using kafka-python library version 2.02. (latest) But it does not have client_rack attribute. But the libs support kafka 2.4.x which has client_rack…
Yejin
  • 541
  • 2
  • 15
  • 32
1
vote
1 answer

AWS Lambda importError: Unable to import module 'lambda_function': No module named 'confluent_kafka.cimpl

I am trying to use confluent_kafka in my AWS Lambda function and I am adding a layer to it. The layer has a .zip file which contains following packages: confluent_kafka confluent_kafka_1.8.2.dist-info confluent_kafka.libs Note: All the above…
Rubin Shah
  • 21
  • 2
1
vote
1 answer

How to handle exceptions in Confluent-Kafka for python

I'm trying to use confluent_kafka to consume some messages from a broker. I'm getting an exception which I fully understand. I'm trying to get earliest offsets for a topic/partition which doesn't exist. I'm trying to handle this exception properly…
ScaryAardvark
  • 2,855
  • 4
  • 30
  • 43
1 2 3
…
14 15