Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-python]

Confluent Kafka Python is a performant implemention of Kafka producers, consumers and the admin client in Python and it is based on librdkafka.

Confluent Kafka Python is a performant implemention of a Kafka client library based on librdkafka. You can use it to connect to Kafka brokers in Python.

219 questions
0
votes
2 answers

How to reread uncommitted messages until they are committed?

Let's say we are using Kafka with manual commits. We are processing the incoming message. But if there is a failure in processing for any reason, we want to reread the message again and we want to ensure that as long as it is not comitted, it will…
Zeruno
  • 1,391
  • 2
  • 20
  • 39
0
votes
1 answer

How to change SubjectNameStrategy and use different schemas in confluent-kafka-python's AvroProducer?

I am using confluent-kafka-python's AvroProducer for serializing. How could I change the SubjectNameStrategy to RecordNameStrategy so that I can use different schemas in the same topic ? Or is there a better way to achieve the same goal ?
ouba
  • 3
  • 3
0
votes
0 answers

Twint to Kafka -> Dont receive any events in my topic

i'm trying to load some tweets via TWINT into a kafka topic with the confluent-kafka[avro] Producer. I don't get any errors but my topic wont receive any events from twint. I even get succes msg, when debugging(with try and except). My Code: import…
Hansanho
  • 295
  • 1
  • 3
  • 13
0
votes
0 answers

Error with avro schema: TypeError: unhashable type: 'dict' Avro Schema

An error occurs while processing the schema, I can not understand what the problem is. Where am I making a mistake? I have this schema: my_schema = """ { "name": "MyClass", "type": "record", "namespace":…
zrfnz
  • 41
  • 6
0
votes
0 answers

How to read data from beginning only once per group in kafka topic?

I'm using confluent-kafka KafkaConsumer library for reading data from kafka. Here is the scenario. Demo topic is compaction enabled. Consumer-1 is reading data using the latest property with a group.id="group-1". Now I want Consumer-2 to read all…
Akash Pagar
  • 637
  • 8
  • 22
0
votes
0 answers

How to consume last N messages (roughly) from a topic of M unbalanced partitions in Kafka?

I have a need to consume last N messages from a Kafka topic. The topic has M partitions, each due to whatever reason has more or less messages than others. In other words, messages are not evenly distributed across partitions. What would be the…
wxh
  • 619
  • 7
  • 20
0
votes
2 answers

Mock an external service instantiated in python constructor

I'm working on unit tests for a service I made that uses confluent-kafka. The goal is to test successful function calls, exception errors, etc. The problem I'm running into is since I'm instantiating the client in the constructor of my service the…
Ryan Hardin
  • 81
  • 2
  • 3
  • 12
0
votes
1 answer

Deserializing JSON data from Kafka stream - Kafka connect to PostgreSQL

I'm streaming topic with Kafka_2.12-3.0.0 on Ubuntu in standalone mode to PosgreSQL and getting deserialization error. Using confluent_kafka from pip package to produce kafka stream in python (works ok): {"pmu_id": 2, "time": 1644329854.08,…
Hrvoje
  • 13,566
  • 7
  • 90
  • 104
0
votes
0 answers

Calculate moving average score from a Kafka topic with Python

I currently have a producer producing a Kafka topic with messages containing average score of, something similar to the below: { "post_id": "post_123", "sentiment_score: "0.6789" } The kafka cluster is being hosted on Confluent cloud and…
knl
  • 969
  • 11
  • 35
0
votes
0 answers

For Kafka (Python), how should the error, the transaction requires abort be handled?

How should the transaction requires abort case be handled for the Transactional Producer API? According to the documentation, if while trying to commit a transaction, an error occurs where a transaction should be aborted (possibly due to…
Moritz
  • 2,987
  • 3
  • 21
  • 34
0
votes
1 answer

Kafka producer automatic fail-over/fail-back

I would like to know if it's possible to configure 2 different Kafka cluster in a Kafka producer. Currently I'm trying to have my producers & consumer failback automatically to a passive cluster without reconfiguring (bootstrap.servers) and…
Astream
  • 11
  • 2
0
votes
0 answers

How can I connect confluent-kafka which installed on openshift?

I deploy confluent-kafka on openshift and I can open control-center on browser. I deploy openshift via route on openshift. I want to connect kafka broker as a consumer on different server. So I deploy another route for kafka broker like this: kind:…
CompEng
  • 7,161
  • 16
  • 68
  • 122
0
votes
0 answers

Serialize Python class using AvroProducer confluent-kafka

I am pretty new to the confluent-kafka and python, just would like to know if there a way in python we could serialize the python class to an kafka message using avro schema. I am currently using AvroProducer provided by confluent-kafka, however, i…
Rachel
  • 349
  • 1
  • 3
  • 19
0
votes
2 answers

AvroSerializer: schema for orderbook snapshots

I have a Kafka cluster running and I want to store L2-orderbook snapshots into a topic that have a dictionary of {key:value} pairs where the keys are of type float as the following example: { 'exchange': 'ex1', 'symbol': 'sym1', 'book':…
CarloP
  • 99
  • 1
  • 12
0
votes
0 answers

Confluent-Kafka python delete_topic getting recreated

When I'm trying to delete a Kafka topic using Confluent-Kafka AdminClient Delete_topic, The topic is getting deleted for the moment but is getting recreated after few seconds. I tried Producer.flush() , Consumer.unsubscribe() to avoid recreating but…
Shubhendra Tomar
  • 1
  • 1
1 2 3
…
14 15