Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-python]

Confluent Kafka Python is a performant implemention of Kafka producers, consumers and the admin client in Python and it is based on librdkafka.

Confluent Kafka Python is a performant implemention of a Kafka client library based on librdkafka. You can use it to connect to Kafka brokers in Python.

219 questions
0
votes
0 answers

confluent-kafka producer does not understand broker

I am trying to set up a simple producer using confluent-kafka-python[1]. I copied the code 1:1, just changing the host and the topic. The code looks like this: producer.py from confluent_kafka import Producer import socket KAFKA_HOST =…
kev
  • 8,928
  • 14
  • 61
  • 103
0
votes
1 answer

Confluent Kafka - Lowest read offset

When consuming from a non-compacted topic, how can I determine the earliest available offset on the topic? In my case the message retention is 7 days. Low watermark does not help in this case because it is 0 and of course not what I am looking for.…
rpd
  • 462
  • 1
  • 9
  • 24
0
votes
1 answer

Is there a way to use kafka consumers and producers in the same Python file using confluent_kafka

I am currently trying to allow for two scripts to communicate in Python using confluent_kafka. The idea is that script 1 produces a message to script 2 for a temperature reading. Script 2 consumes the message from s1 and produces the current…
MichaelLong102
  • 35
  • 5
0
votes
1 answer

Not getting old messages even with 'auto.offset.reset': 'earliest'

I'm trying to write a kafka consumer in python using Confluent Kafka. I can get all new messages but I don't get any old messages if I kill and restart my consumer def confluent_kafka_consumer(app): with app.app_context(): import config …
user14530855
  • 1
  • 1
0
votes
1 answer

how to send the 'auto.register.schemas':'false' in python producer

I'm recieveing the below error when try to produce to the kafka topic. Error connecting to Kafka topic KafkaError{code=_VALUE_SERIALIZATION,val=-161,str="User is denied operation Write on Subject: xxxxx-value (HTTP status code 403, SR code…
Pumudu Fernando
  • 11
  • 4
0
votes
1 answer

The last messages are not consumed from a kafka topic even though they should be, leaving a constant consumer lag

I have a very basic kafka consumer which needs to consume data from a 32-partitions topic with a large amount of data on each partition. It manages to consume most data from that topic, but once we get towards the end of each partition, it does not…
Florentin Hennecker
  • 1,974
  • 23
  • 37
0
votes
0 answers

python 3.8 and above - No module named 'confluent_kafka.cimpl'

After migrating python version from 3.7 to 3.8/3.9, seeing an issue with Kafka confluent compatibility in one of our Lambda functions. I am using Kafka confluent 1.6 library. Error while running lambda: { "errorMessage": "Unable to import module…
devoverflow
  • 25
  • 4
0
votes
1 answer

Is there a suitable python library for doing stream processing with Kafka topics?

I am trying to find a suitable Python library to do stream processing with streams Kafka topics, Kafka streams. Specifically, I am looking for libraries that support the following operations. KStream-to-Kstream KTable-to-Ktable KTable-to-KTable…
Eagle1992
  • 3
  • 1
0
votes
1 answer

How do I tell a topic on confluent cloud to use a specific schema programmatically?

So I know how to create topics on Confluent Cloud with the confluent_kafka AdminClient instance but I’m not sure how to set the topic’s message schema programmatically? To clarify, I have the schema I want to use saved locally in an avro schema…
user1943079
  • 511
  • 6
  • 20
0
votes
0 answers

Queuing full with confluent kafka

I'm using websockets as producers that are kafka connected (using the confluent_kafka library) to a postgresql database. I have 4 parallel websockets running in different scripts, connected to different topics which output to different tables in the…
Kafkaesque
  • 37
  • 3
0
votes
0 answers

How to avoid WORKER TIMEOUT in python?

I have written a consumer to read messages from kafka topics, in the python code I have written logic inside infinite loop to poll message from kafka topic at every second. However, we are getting WORKER TIMEOUT issue and server get restarted…
Ashvini Dhainje
  • 1
0
votes
0 answers

Cunfluent Kafka (Python) - Check partition leader

The topic I am consuming data is distributed in 3 brokers and 18 partitions. Each broker is the leader for 6 partitions. How can I list the leader of each partition on my Python application? (I am using confluent kafka library). I need to do this…
rpd
  • 462
  • 1
  • 9
  • 24
0
votes
1 answer

Connect Kafka to Neo4j

I have created a simple python code that generates user_id, receipent_id and amount. I have created a kafka producer and consumer. Python code returns data as a json. Now I am trying connect my data to Neo4j through kafka but I am unable to do…
Nekodu
  • 1
  • 1
0
votes
0 answers

VSCode doesn't recognize ConfluentKafkaLibrary to use in Robot Framework

I need to do a test automation in Robot Framework using Kafka topics. I'm trying to use the ConfluentKafkaLibrary library, but VSCode doesn't recognize it. I already installed the commands: pip install robotframework-confluentkafkalibrary pip…
Alexandre de Paula Sousa
  • 1
  • 1
0
votes
0 answers

Error while building confluent-kafka in docker for alphine image

I am trying to build a python application which require confluent-kafka package. But while building in bamboo, I got the below error fatal error: librdkafka/rdkafka.h: No such file or directory build 13-Dec-2022 11:46:59 23 | #include…
potterson11
  • 147
  • 7
1 2 3
…
14 15