Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-python]

Confluent Kafka Python is a performant implemention of Kafka producers, consumers and the admin client in Python and it is based on librdkafka.

Confluent Kafka Python is a performant implemention of a Kafka client library based on librdkafka. You can use it to connect to Kafka brokers in Python.

219 questions
2
votes
1 answer

Not receiving messages from Kafka Topic

I am receiving None when calling poll() in this program but I am getting the messages when running the kafka-console-consumer.bat from cmd, I can't figure out what exactly the problem. The execution starts from main.py from queue import Queue from…
PanicLion
  • 187
  • 1
  • 9
2
votes
3 answers

Cannot import Producer from confluent_kafka

I'm using conda environment with python 3.9 and confluent_kafka library (installation by pip install confluent-kafka). I have also installed librdkafka. From pycharm I cannot import: from confluent_kafka import Producer I receive this…
Krzysiek
  • 31
  • 1
  • 6
2
votes
0 answers

How to add configuration setting for sasl.mechanism PLAIN (API) and GSSAPI (Kerberos) authentication in python script

Need some help to set the configuration for sasl.mechanism PLAIN (API) and GSSAPI (Kerberos) authentication. We are using confluent Kafka here, there are two scripts, one a python script and the second one is a bash script which calls the python…
sparsh bhardwaj
  • 93
  • 2
  • 8
2
votes
1 answer

Getting error "confluent_kafka.libs are not found" when python programs are converted to exe application using pyinstaller(4.5) and librdkafka 1.7.0

The latest version of librdkafka 1.7.0 when used in a python(python 3.8.11) program on windows 64 bit os works fine but when I convert the program to a exe file and run I am getting the following error File "produce-simple-avro.py", line 1, in File…
Harshith Yadav
  • 106
  • 1
  • 9
2
votes
1 answer

When does a delivery callback called with an error in confluent kafka client?

The Confluent Kafka library (python version in this case) has a produce method which takes a delivery callback function: kafka_producer.produce(topic=topic, key=key, value=value, …
coderelliot
  • 421
  • 5
  • 15
2
votes
0 answers

kafkashaded.org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms

I am currently working on the use case, where I am writing the pyspark dataframe to confluent-kafka topic. def write_data(rows): rows.selectExpr("to_json(struct(*)) AS value") \ .write \ .format("kafka") \ .option("kafka.bootstrap.servers",…
Ajay Kumar
  • 586
  • 5
  • 21
2
votes
1 answer

How to read and process high priority messages in kafka consumer?

Is there any method to process messages with high priority first? I tried creating three topics 'high', 'medium', and 'low', and subscribed to all three topics with one consumer and if there is an unprocessed message in the 'high' topic it will…
Sudheer
  • 23
  • 3
2
votes
1 answer

How to do I connect Kafka python to accept username and password for jaas like it's done in Java?

Using an existing working Java example I am trying to write a python equivalent of the producer using python-kafka and confluent_kafka library. How do I configure sasl.jass.config in python with the information like that in Java below? import…
codeBarer
  • 2,238
  • 7
  • 44
  • 75
2
votes
0 answers

AWS MSK setting advertised listeners

I am trying to set the advertised.listeners config of my MSK cluster. However, I am getting the following error back: requirement failed: inter.broker.listener.name must be a listener name defined in advertised.listeners I am not sure what this…
prats
  • 225
  • 5
  • 15
2
votes
1 answer

How to decrypt custom encrypted message in Python Confluent_Kafka using key.deserializer and value.deserializer?

My team is using confluent_kafka to produce real time streaming and is being pushed to a queue using custom encryption. The technology behind is Java. I am trying to consume the messages using python from the topic and able to do so but unable to…
Mayank Ahuja
  • 29
  • 3
1
vote
1 answer

Kafka ensuring consumer group stays alive

I have a process which spawns a producer and consumer in separate pods (kubernetes). I want to use auto.offset.reset "latest", and thus need to guarantee that the consumer pod spins up prior to the producer pod, as I do not want the producer to…
bbd108
  • 958
  • 2
  • 10
  • 26
1
vote
1 answer

I'm not receiving the keys in the Kafka content - Python/confluent

I'm trying to receive the message from the topic using Python script. from confluent_kafka import Consumer, KafkaError import uuid # Kafka broker details broker = "something" topic = "something" group = str(uuid.uuid4()) # Kafka consumer…
Aleksandar
  • 84
  • 1
  • 4
1
vote
0 answers

Python: confluent-kafka (librdkafka) producer Timeout error

I've got a 9 nodes kafka cluster hosted on AWS MKS and I'm using confluent-kafka library with python. While producing to a topic I get too many Timeout errors like: %5|1684850550.061|REQTMOUT|eb3004f09ce7#producer-1| [thrd:sasl_ssl://broker1.a]:…
JustAnOtherNickname
  • 11
  • 3
1
vote
0 answers

Reset offsets to earliest using confluent_kafka without consuming

I am writing a script to reset the offsets of a reused consumer group to the start of a topic. This is the command I am trying to simulate from the client side using python and confluent_kafka: bin/kafka-consumer-groups.sh --bootstrap-server…
user4446237
  • 636
  • 8
  • 21
1
vote
0 answers

Kafka - Consumer failed to start: Connection refused

I tried to produce some json data to multinode kafka broker using confluent-kafka-python When I execute the python script, an error message occur that said > %3|1680100333.779|FAIL|rdkafka#producer-1| [thrd:100.25.177.77:9095/bootstrap]:…
humanlearning
  • 63
  • 1
  • 6
1 2
3
…
14 15