Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-python]

Confluent Kafka Python is a performant implemention of Kafka producers, consumers and the admin client in Python and it is based on librdkafka.

Confluent Kafka Python is a performant implemention of a Kafka client library based on librdkafka. You can use it to connect to Kafka brokers in Python.

219 questions
0
votes
0 answers

Kafka Confluent Python SSL truststore configuration

I have a confluent Kafka consumer code using Python. I also have the truststore ca files including: certificate.jks and chain_certificate.pem. I know that we I am not able to use .jks file in python code and the PEM file that I have is a chain file…
Ali Lordifar
  • 87
  • 4
0
votes
1 answer

Confluent Kafka poll, when does a message get committed

I have a Python application that has autocommit=True and it is using poll() to get messages with a interval of 1 second. I was reading on the documentation and it mentions that polling reads message in a background thread and queues them so that the…
Miguel Wang
  • 183
  • 1
  • 12
0
votes
1 answer

Confluent-Kafka Python - Describe consumer groups (to get the lag of each consumer group)

I want to get the details of the consumer group using confluent-kafka. The cli equivalent of that is ` ./kafka-consumer-groups.sh --bootstrap-server XXXXXXXXX:9092 --describe --group my-group My end goal is to get the value of lag from the…
Taimoor Abbasi
  • 51
  • 6
0
votes
1 answer

Unable to get kafka connect redshift connector working

Following the question and suggestions here: Kafka JDBCSinkConnector Schema exception: JsonConverter with schemas.enable requires "schema" and "payload", I am trying to sink records into redshift using redshift connector and a producer written in…
Akshay Baura
  • 41
  • 1
  • 6
0
votes
1 answer

cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="No such configuration property: "bootstrap_servers""}

I am trying to produce AVRO data to Kafka topic through python producer script, I already installed python dependencies avro-python3 and confluent_kafka, however when running this script i got below error: File "./kafka_producer_avro.py", line…
Amira Hussein
  • 13
  • 4
0
votes
1 answer

KafkaSource connection to Confluent Kafka (with SSL & SchemaRegistry)

I tried to connect to Confluent Kafka with KafkaSource (from MLRun) and I used historically this easy code: # code with usage 'kafka-python>=2.0.2' from kafka import KafkaProducer, KafkaConsumer consumer = KafkaConsumer( 'ak47-data.v1', …
JIST
  • 1,139
  • 2
  • 8
  • 30
0
votes
0 answers

Unable to access Kafka broker inside docker network

On my repository I have run docker-compose up -d on the root directory and I am checking the myapp container logs. The producer and consumer Python files are unable to connect to the Kafka broker, prompting the error: Failed to resolve…
Martim Sousa
  • 7
  • 2
0
votes
1 answer

Get consumer group state using confluent kafka python

I am doing this way currently to get the groupmetadata for list of consumers admin = AdminClient({ 'bootstrap.servers' : config['kafka']['brokers'] }) for group in config['kafka']['groups']: metadata = admin.list_groups(group) …
Gibbs
  • 21,904
  • 13
  • 74
  • 138
0
votes
0 answers

Kafka consumer does not read transactional messages

My Kafka consumer does not read commited transactions from Kafka topic, when 'isolation.level': 'read_committed' is set. Customer, writen in Python, is using confluent-kafka module. Producer is Spring boot application. Confluent Kafka cluster has…
Robert Szomolanyi
  • 1
  • 1
0
votes
1 answer

Can´t consume messages from topic

I have the following code: from confluent_kafka.admin import AdminClient, NewTopic a = AdminClient({'bootstrap.servers': 'localhost:9092'}) new_topics = [NewTopic(topic, num_partitions=3, replication_factor=1) for topic in ["topic1", "topic2"]] fs…
Data Mastery
  • 1,555
  • 4
  • 18
  • 60
0
votes
0 answers

Difference between Consumer and Deserializing consumer

There are two types of consumers provided in the confulent kafka python API: Consumer and the Deserializing Consumer and I have got a few questions. Is deserializing consumer suited more for avro schemas? Are there any performance difference…
thegreatcoder
  • 2,173
  • 3
  • 19
  • 28
0
votes
0 answers

LocalBuffer full error while writing to topic in Kafka

So, we have a producer which was running fine for a few months with no errors but has suddenly started erroring out with 'BufferError: Local: Queue full' I have had this issue initially, and then by going through documentation and S.O posts ,…
Saugat Mukherjee
  • 778
  • 8
  • 32
0
votes
0 answers

Kafka use seek to manually manage offset, consumer lose some messages

I have a script to test for at least one consume the producer import json import random import time from confluent_kafka import Producer import config p = Producer({'bootstrap.servers':','.join(config.KAFKA_HOST),}) total_count = 0 c = 0 try: …
dakang
  • 1
  • 2
0
votes
0 answers

Python protobuf how to consume and produce to different topics

I am currently working on a project where I need to connect multiple micro-services with Kafka and to serializer/deserializing messages with protobufer. The services are written with python and use python package "protobuf" (confluent-kafka). I am…
ItamarYard
  • 1
  • 1
0
votes
0 answers

How to implement exactly-once on Apache Kafka using Python?

I am trying to implement an exactly-once concept using confluent kafka library. But I couldn't succeed. Here is my producer code: from confluent_kafka import Producer, SerializingProducer import socket, json, random, time conf =…
Yunus Emrah Uluçay
  • 129
  • 9
1 2 3
…
14 15