Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-python]

Confluent Kafka Python is a performant implemention of Kafka producers, consumers and the admin client in Python and it is based on librdkafka.

Confluent Kafka Python is a performant implemention of a Kafka client library based on librdkafka. You can use it to connect to Kafka brokers in Python.

219 questions
1
vote
1 answer

Read Data from Kafka Broker at Specific Time Intervals (Python)

I have a system where a machine is reading data and continuously appends a .txt file. This dataset is read into a Kafka broker via Kafka Connect and would then be pre-processed with some Python code. The machine operates roughly every 5 mins, so we…
Aaron Gonzalez
  • 86
  • 11
1
vote
0 answers

How to handle ValueDeserializationError in confluent_kafka python?

This is the base consumer class I'm using for creating new consumers. It works fine for "enable.auto.commit":True consumer. But when I create a consumer with enable.auto.commit=False and any of the (KeyDeserializationError,…
Akash Pagar
  • 637
  • 8
  • 22
1
vote
0 answers

Confluent Kafka: Setting Retention.ms for an individual topic not working as expected

I'm using confluent Kafka in my project where the messages sent to a particular topic need to be deleted after a retention time. So I set retention.ms for the individual topics but it's not working (Still I can see messages after its retention…
K Surya Kumar
  • 65
  • 8
1
vote
0 answers

Confluent-kafka consumer not reading last published message

I have a confluent-kafka consumer which will be running once in a day. while its running i just wants to get one last message from the topic. I have below code, but is not fetching the last message, just reading the messages other than the last…
kaviya .P
  • 469
  • 3
  • 11
  • 27
1
vote
0 answers

Unable to catch confluent-kafka-python timeout errors

My confluent-kafka-python - version 1.7.0 producer code is as below from confluent_kafka import KafkaError, KafkaException def publish_data(): try: producer.produce( topic=topic, value={"produced_epoch":…
Zaks
  • 668
  • 1
  • 8
  • 30
1
vote
1 answer

Jupiter Docker: Using pip install in a child docker image

I was trying to run a custom version of a Jupiter notebook image on MacOS, just wanted to install a confluent-kafka library in order to use the kafka python client. I followed the simple instruction provided in the docs. This is the Dockerfile: FROM…
Cr4zyTun4
  • 625
  • 7
  • 18
1
vote
0 answers

" fatal error C1083: Cannot open compiler generated file: '': Invalid argument" while trying to "pip install robotframework-confluentkafkalibrary"

I am trying to install robotframework-confluentkafkalibrary using command pip install robotframework-confluentkafkalibrary during which i get the following error : C:\Program Files (x86)\Microsoft Visual…
harmanjit singh
  • 21
  • 2
1
vote
1 answer

Python confluent_kafka - List All Consumer currently Listening to a Topic

How to get of consumer.id which are listening to a topic using Confluent_Kafka? I am able to get the consumer-groups and topic. But not the consumer name/id.
Marshall
  • 127
  • 3
  • 11
1
vote
1 answer

How to check number of partitions in Kafka(confluent_kafka)

I am trying to do batch etl by using confluent_kafka Python Packages at 0 0 * * * everday. I know that there are 4 partitions in my stream but it could be changed so Is there any way to check total number of partitions in specific topic? My consumer…
mjeday
  • 78
  • 6
1
vote
1 answer

Cannot create local docker-compose file for Confluent Cloud Setup

I want to create a local kafka setup using docker-compose that replicates very closely the secured kafka setup in confluent cloud. The cluster I have in Confluent Cloud can be connected to using c = Consumer( { "bootstrap.servers":…
Nilan Saha
  • 191
  • 1
  • 9
1
vote
1 answer

ProduceResponse from confluent cloud broker using confluent-kafka-python

This log is available with debug=all config in confluent-kafka-python producer client for confluent cloud. How to parse ProduceResponse for rtt? Can it be done using on_delivery callback? |RECV|rdkafka#producer-1|…
Darkwing Duck
  • 11
  • 1
1
vote
0 answers

How to detect end of partition with kafka-python?

I want to try the kafka-python package instead of the Confluent Kafka python binding because I want a pure Python package and only kafka-python does that. However, I am unable to work out how to implement a particular behaviour that I depend on in…
Anders Lindstrom
  • 185
  • 2
  • 4
1
vote
2 answers

Reading from latest offset from kafka with python

I am using the confluent-kafka Python library to read from kafka. I am using the following consumer settings Consumer ={ "bootstrap.servers" : kafka_server, "group_id" : "testing", "auto.offset.reset" : "latest"} My goal is to ensure that I am…
PRATIK CHAPADGAONKAR
  • 19
  • 1
  • 5
1
vote
1 answer

Schema evolution when adding new field

Imagine there are to separate apps: producer and consumer. The code of producer: import os from confluent_kafka import avro from confluent_kafka.avro import AvroProducer avsc_dir = os.path.dirname(os.path.realpath(__file__)) value_schema =…
Альберт Александров
  • 807
  • 1
  • 15
  • 27
1
vote
1 answer

confluent_kafka.error.ValueSerializationError: KafkaError{code=_VALUE_SERIALIZATION,val=-161 : ValueError

I am new bee to python and trying to use 'confluent_kafka' for avro message produce. Using 'confluent_kafka.schema_registry.avro.AvroSerializer' for the same (referred :…
MHegde
  • 329
  • 3
  • 14
1 2 3
…
14 15