Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-python]

Confluent Kafka Python is a performant implemention of Kafka producers, consumers and the admin client in Python and it is based on librdkafka.

Confluent Kafka Python is a performant implemention of a Kafka client library based on librdkafka. You can use it to connect to Kafka brokers in Python.

219 questions
0
votes
0 answers

kafka producer docker app is failing on windows docker desktop, however it works fine in MAC

I am working on kafka producer docker app which is working fine on MAC however it is failing on windows and Linux machines with the following error. The docker build is working on windows/linux/mac however when we run docker run, the apps seems to…
bomsabado
  • 35
  • 6
0
votes
0 answers

confluent_kafka python error "Unsupported value "SASL_SSL" for configuration property "security.protocol": OpenSSL not available at build time"}

I am writing a Avro Python Producer and when I try to run the code I get this error: cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Unsupported value "SASL_SSL" for configuration property "security.protocol": OpenSSL not available…
rb_9999
  • 5
  • 4
0
votes
1 answer

Setting processing.guarentee to exactly_once creates deserialization error at python consumer kafka

I have java Kafka stream processing application and python application. Java application produces data and python consumer consumes it. When the processing.guarantee is set to exactly_once, then the python consumer is not able to deserialize the…
pacman
  • 725
  • 1
  • 9
  • 28
0
votes
1 answer

0x4 Decompression error in Python Confluent Kafka

I've been having trouble getting python-confluent-kafka to work on my windows server. When creating a simple consumer on my local machine, everything works fine. However, once on the windows server, I will receive the messages but get the following…
Oskar
  • 83
  • 5
0
votes
1 answer

how can I fix TypeError: can't set attributes of built-in/extension type 'cimpl.Consumer'

example.py def simple(): msg = consumer.poll(timeout=int(timeout)) if msg is None: break if msg.error(): if (msg.error().code() == KafkaError.UNKNOWN_TOPIC_OR_PART): response_code = 409 …
PRAJAKTA DHIVAR
  • 7
  • 2
0
votes
1 answer

confluent-kafka consumer working but not producer

Using confluent-kafka, I'm trying to work out with Amazon MSK. It seems to connect to MSK since consumer is working but producer is not producing anything. I checked with kafka-console-producer.sh that consumer module is working correctly. We have…
ChrisNex
  • 1
  • 1
0
votes
1 answer

Error installing confluent kafka - ld: library not found for -lrdkafka

I am trying to install confluent kafka python module on my mac os. My python version is 3.9 I am keep getting below error. Does anyone know how to fix this? clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic…
user4981459
0
votes
1 answer

Unable to push subsequent transactions to Kafka Producer

I'm trying to send events to Kafka in batches (let's say 10 events in a batch) to maintain transactional guarantees. But I'm able to successfully send only the first batch of events to Kafka. All subsequent batches fail with the reason…
Shivam
  • 11
  • 3
0
votes
1 answer

Python Confluent-kafka DeserializingConsumer is not reading the messages from Kafka topic, even though the messages are present

I have a topic with avro schema, I am producing the messages via Python code and it works completely fine. When I consume the messages from CLI, I can consume them successfully without errors. When I am trying to consume via Python code, it prints…
Ashish Ahuja
  • 85
  • 1
  • 8
0
votes
0 answers

Can Kafka Consumer and Producer be on same thread?

Description I am running a Kafka server on my computer with this docker-compose file. I have created basic consumer and producer applications which I run simultaneously, and I can send and receive messages through a topic as expected. I have also…
Skogis
  • 1
  • 1
0
votes
1 answer

Kafka Producer Function is not producing messages to Kafka via Google Cloud Functions

I am using GCP with its Cloud Functions to execute web scrapers on a frequent basis. Also locally, my script is working without any problems. I have a setup.py file in which I am initializing the connection to a Kafka Producer. This looks like…
ku11
  • 43
  • 5
0
votes
1 answer

How to slow down kafka consumer when there is load on the DB and vice versa

If a lot of message are produced on kakfa and if I try to consume (using python confluent_kafka library) and process them, the database (working on mysql DB) gets loaded with a lot of queries quickly. I want to slow down the consuming speed based on…
DeadLock
  • 1
0
votes
0 answers

How to capture Application maximum poll interval exceeded error with confluent-kafka-python?

My microservice uses confluent-kafka-python. Once in a while it fails with this error %4|1654121013.314|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 67ms (adjust max.poll.interval.ms for…
wxh
  • 619
  • 7
  • 20
0
votes
0 answers

confluent-kafka python: List topics of consumer group

With the aim of getting consumer group to topic mapping, similar to that in Kafka CLI, I was trying to use the describe_config api offered by AdminClient ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group testing-group…
Tushar
  • 528
  • 4
  • 20
0
votes
1 answer

Python client for Managing Kafka ACL

Do we have any python client for managing Kafka ACL? I know we have Java Admin client and same can be performed using Kafka scripts (kafka-acl.sh) but cannot find one for python client. confluent python library doesn't support it. Came across…
Vikash Mishra
  • 123
  • 2
  • 10
1 2 3
…
14 15