Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-python]

Confluent Kafka Python is a performant implemention of Kafka producers, consumers and the admin client in Python and it is based on librdkafka.

Confluent Kafka Python is a performant implemention of a Kafka client library based on librdkafka. You can use it to connect to Kafka brokers in Python.

219 questions
0
votes
1 answer

AWS lambda to Confluent Cloud latency issues

I am currently using basic version of cluster on Confluent cloud and I only have one topic with 9 partitions. I have a REST Api that’s setup using AWS lambda service which publishes messages to Kafka. Currently i am stress testing pipeline with…
icyanide
  • 31
  • 5
0
votes
1 answer

kafka python - What is the right way to find new kafka topics that a consumer has not been yet subscribed to?

I am new to the kafka world and trying to do the following for a kafka consumer in python get a list of all kafka topics. get a list of topics a consumer has subscribed to. subscribe to new topics (that have not yet been subscribed to). Note: I am…
rocky502
  • 23
  • 6
0
votes
1 answer

Kafka Latency is linearly increasing

I'm trying benchmark Apache Kafka and at the moment i am trying to measure the latency. In order to do this, i am producing 10 MB of tweet data (~14500 tweets) to not lose an overview about whats going on, but i can basically send arbitrarily much…
Manu
  • 284
  • 2
  • 20
0
votes
1 answer

Apache Kafka & JSON Schema

I am starting to get into Apache Kafka (Confluent) and have some questions regarding the use of schemas. First, is my general understanding correct that a schema is used for validating the data? My understanding of schemas is that when the data is…
vibuc
  • 3
  • 2
0
votes
1 answer

Kafka consumer consuming only 1 recent record

I am running the code, it is consuming only the 1 most recent record from the topic. I want all the records of the topic. Also the output file in which I am storing the record has the record's metadata as well. How can I skip the metadata. from…
Monika Upwanshi
  • 1
  • 1
0
votes
0 answers

Error installing confluent-kafka on Cloud Dataflow

Hope you're all doing well. I'm trying to deploy a Dataflow pipeline that will ingest data from Kafka. To do that, I'm using the lib confluent-kafka, but I'm getting the error below. NameError: name 'confluent_kafka' is not defined I'm already…
Giorge Caique
  • 199
  • 1
  • 1
  • 16
0
votes
1 answer

How to overwrite default config values in kafka for confluent-python client?

As a beginner, I'm exploring Apache Kafka and confluent-kafka-python clients. When I tried to send simple messages from the producer, the consumer was able to consume messages successfully. Thought that I would give it a try for sending the image as…
AkshayAwate
  • 3
  • 2
0
votes
1 answer

ksqlDB Querying at Best Practice

I am currently building a ksqlDB instance and I target to deploy it in interactive mode. I created streams and table to serve windowed aggregations with RocksDB. I would like to query the cache with REST API calls (i.e. Python wrapper for KSQL Rest…
Soner Guzeloglu
  • 103
  • 1
  • 11
0
votes
1 answer

Trouble with confluent-kafka consumer working with asyncio

I am trying to integrate asyncio functionality into my kafka topic listener and am running into issues (pretty new to async programming in python). I have a confluent-kafka consumer created that is listening to a topic. The topic has messages…
StormsEdge
  • 854
  • 2
  • 10
  • 35
0
votes
1 answer

Confluent Schema Registry - AvroProducer - Connection Error

I am trying to produce messages in Avro format using AvroProducer class of confluent_kafka. Kafka and Schema-Registry run as cluster of 3 nodes in the same network. I am reading the schema from string and initializing AvroProducer as…
meliksahturker
  • 922
  • 2
  • 11
  • 20
0
votes
0 answers

Kafka producer with python multi threading

I'm planning to build a critical component which produces the messages to kafka. I'm just thinking, is there any way that python multithreading will help us in writing the efficient kafka producer with asyncio, multithreading ? Also here I'm…
Learnis
  • 526
  • 5
  • 25
0
votes
1 answer

Confluent Kafka producer message format for nested records

I have a AVRO schema registered in a kafka topic and am trying to send data to it. The schema has nested records and I'm not sure how I correctly send data to it using confluent_kafka python. Example schema: *ingore any typos in schema (real one is…
rayyyyychul
  • 35
  • 1
  • 6
0
votes
1 answer

Using confluent-kafka-python to setup schema for topic

I am new to confluent-kafka-python i am running kafka using docker-compose from cp-all-in-one. I created a producer producing data to a new topic same as the offical tutorial. but I can not find any information regarding how to set up the schema or…
sam
  • 335
  • 2
  • 5
  • 19
0
votes
1 answer

Specifying replication factor in Confluent Kafka Python Client

I have setup a Single broker instance of Kafka along with Zookeeper, Kafka-tools,Schema-registry and control-center.The setup is done using docker compose and using the Confluent provided images.Here is how the docker-compose looks like: …
AR7
  • 366
  • 1
  • 16
0
votes
0 answers

Python 3 Confluent Kafka Running Producer and Consumer within single function

I am trying to run Producer and Consumer within single function. The flow is below : message arrived --> consume from topic 1 --> transform --> produce to topic 2 If I instantiate Producer and consumer in this pattern (Producer first then…
dataplumber
  • 375
  • 3
  • 16
1 2 3
…
14
15