Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-python]

Confluent Kafka Python is a performant implemention of Kafka producers, consumers and the admin client in Python and it is based on librdkafka.

Confluent Kafka Python is a performant implemention of a Kafka client library based on librdkafka. You can use it to connect to Kafka brokers in Python.

219 questions
1
vote
1 answer

confluent-kafka-python library: read offset per topic per consumer_group

Due to pykafka EOL we are in the process of migration to confluent-kafka-python. For pykafka we wrote an elaborated script that produced output in the format: topic consumer…
Dan M
  • 1,175
  • 12
  • 23
1
vote
2 answers

Getting last log value with Structlog and passing it as a variable to a function

Currently I am trying to implement a function call that sends failed messages from a converter to a DLQ topic with Kafka. As part of the DLQ message I want to include the exception error that we also log. the code: except…
Harvey
  • 668
  • 2
  • 7
  • 15
1
vote
0 answers

Kafka: Understanding Broker failure

I have a Kafka cluster with: 2 brokers b-1 and b-2. 2 topics with both: PartitionCount:1 ReplicationFactor:2 min.insync.replicas=1 Here is what happened: %6|1613807298.974|FAIL|rdkafka#producer-2|…
max
  • 11
  • 3
1
vote
0 answers

confluent-kafka-python didn't catch timed out metadatarequest in flight in producer

Hi our kafka has crashed for about 2 days, while that one of our producers using confluent kafka for python still tried producing a messages and while that we received logs that say Timed out MetadataRequest in flight and Metadata request failed:…
dfhyr123
  • 11
  • 1
1
vote
1 answer

AvroConsumer from confluent_kafka raises "'dict' object has no attribute 'get_by_id'"

My AvroConsumer from module confluent_kafka.avro always raise 'dict' object has no attribute 'get_by_id' when polling. Although, when I poll with a simple Consumer from confluent_kafka I get the binary serialized. The ccloud CLI also works perfectly…
felicienb
  • 83
  • 10
1
vote
1 answer

Kafka Consumer not consuming from last commited offset after restart

I have a consumer polling from subscribed topic. It consumes each message and does some processing (within seconds), pushes to different topic and commits offset. There are totally 5000 messages, before restart - consumed 2900 messages and committed…
krishna reddy
  • 295
  • 2
  • 15
1
vote
0 answers

Kafka Consumer lag is not zero

I have a Kafka cluster with 40 partitions and I published a few million messages. I have a pool of consumers and for the most part the consumer lag < 100. After my experiment, I stopped publishing any new message and left consumers to drain. But, I…
Icsahn
  • 11
  • 2
1
vote
0 answers

Data loss Producer->Kafka->Nifi

I write messages to kafka from csv files. My producer says that all data produced to Kafka topic. Along with that, I use apache nifi as consumer to kafka topic (ConsumeKafka_2_0 processor). If I produced data to kafka into one stream - all is OK,…
Grigory Skvortsov
  • 431
  • 4
  • 22
1
vote
0 answers

multiprocessing Python Kafka consumer client not getting back messages

a Python consumer client worked fine when ran as standalone but not able to retrieve message when run as a multiprocessing worker with same configuration. The clients always print the message in the block where it get None for msg. Really appreciate…
techie11
  • 1,243
  • 15
  • 30
1
vote
2 answers

Some Python Confluent Kafka Consumers are staying idle/unassigned even though others overloaded/over-assigned

Setup: 120 python confluent-kafka consumers which are all making a subscription to the same set of topics 8 topics with different number of partitions: 1 topic with 84 partitions, several topics with 40-50 partitions, and the rest with 1-10…
Dako Dakov
  • 21
  • 1
  • 6
0
votes
0 answers

Kafka error - 'Disconnected while requesting ApiVersion

I have a Kafka docker set up that I was able to run properly with the following docker-compose.yml. > version: '2' > > services: > zookeeper: > image: bitnami/zookeeper:3.8 > ports: > - 2181:2181 > …
kta
  • 19,412
  • 7
  • 65
  • 47
0
votes
0 answers

Python, apache beam, kafka and bigquery

I use python and apache beam to read streaming data from kafka and insert the data to big query table but I want to insert the data in batches instead of streaming way. I tried to set pipeline streaming mode to True and add batch size to…
muertto
  • 1
0
votes
0 answers

Facing connection issues within confluent-kafka in docker compose environment

I’m relatively new to Kafka and currently trying to set up a Kafka consumer within a Docker Compose environment. Unfortunately, I’ve hit a roadblock with connection issues and I’d greatly appreciate your expertise in helping me troubleshoot this…
cole
  • 75
  • 5
0
votes
0 answers

How to set multiple schemas for topic using confluent_kafka python client?

I'm using confluent_kafka==2.2.0 in my python project. I want to set multiple schemas for a topic. I'm reading docs for AvroSerializer, it says that it has config options to set RecordNameStrategy (see docs here…
Ivan Priz
  • 1
0
votes
0 answers

Data stream with confluent-kafka python giving error - TypeError: You must pass either str or Schema

I am trying to build Stream-Table Join Operations in Kafka python and below is the code that performs join operation on data sent from producer stream_table_join.py from confluent_kafka import DeserializingConsumer, SerializingProducer from…
Polymath
  • 328
  • 1
  • 9
1 2 3
…
14 15