Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
2
votes
1 answer

kafka-python: Connection reset during recv when using SASL_SSL + SCRAM-SHA-512

I am using kafka-python to connect to Kafka Cluster using SASL consumer = KafkaConsumer(bootstrap_servers=['fooserver1:9092', 'fooserver2:9092'], client_id='foo', api_version=(2,2,1), security_protocol='SASL_SSL', sasl_mechanism='SCRAM-SHA-512',…
NinjaTurtle
  • 93
  • 1
  • 6
2
votes
1 answer

How to read and process high priority messages in kafka consumer?

Is there any method to process messages with high priority first? I tried creating three topics 'high', 'medium', and 'low', and subscribed to all three topics with one consumer and if there is an unprocessed message in the 'high' topic it will…
Sudheer
  • 23
  • 3
2
votes
1 answer

How to do I connect Kafka python to accept username and password for jaas like it's done in Java?

Using an existing working Java example I am trying to write a python equivalent of the producer using python-kafka and confluent_kafka library. How do I configure sasl.jass.config in python with the information like that in Java below? import…
codeBarer
  • 2,238
  • 7
  • 44
  • 75
2
votes
1 answer

Kafka-python producer enable.idempotent

i'm facing the duplication data in kafka so i want to use idempotent in kafka-python but unfortunately its not the part of it, how i can stop the duplication data in same partition with out idempotent or there is any way to import it or customize…
farhan pirzada
  • 187
  • 8
2
votes
3 answers

Using python compiled protobuf pb2 as key and value serializer

I am trying to read data from a kafka topiv which has been serialized using google's protobuf. I compiled the proto files using protoc which generated pb2 files. Now i am trying to use faust and create a stream processor but i can't find the correct…
Yuv Joodhisty
  • 71
  • 1
  • 4
2
votes
1 answer

How to consume message from kafka which was produced by kafka-python?

I would like to get help with kafka. Is it possible to consume utf8(Japanese) messages using kafka-console-consumer.sh which was produced by kafka-python ? kafka-python code self._client = KafkaProducer(\ …
Yu Watanabe
  • 621
  • 4
  • 17
2
votes
2 answers

How to connect Kafka consumer to Django app? Should I use a new thread for consumer or new process or new docker container?

I have Django app which should consume Kafka messages and handle it with my handlers and existing models. I use https://kafka-python.readthedocs.io/en/master/usage.html library. What is the right way to connect KafkaConsumer to Django app. Should I…
VladimirProM
  • 31
  • 4
2
votes
0 answers

Reading from Kafka hangs

I'm reading from Kafka but the consumer hangs: def produce(num, producer, topic_name): print("hello2") for i in range(num): future = producer.send('my-topic', bytearray("Hello Kayla 00", "utf-8")) try: …
Jenia Ivanov
  • 2,485
  • 3
  • 41
  • 69
2
votes
1 answer

I want to use kafka producer with python beautifulsoup to send message to kafka broker

I am using kafka-python and BeautifulSoup to Scrape website that I enter often, and send a message to kafka broker with python producer. What I want to do is whenever new post is uploaded on website (actually it is some kind of community like…
ethany21
  • 49
  • 1
  • 7
2
votes
2 answers

kafka-python : avro.io.SchemaResolutionException: Can't access branch index 55 for union with 2 branches

I am using kafka-python 2.0.1 for consuming avro data. Following is the code I have tried: from kafka import KafkaConsumer import avro.schema from avro.io import DatumReader, BinaryDecoder import io schema_path="schema.avsc" schema =…
sgmbd
  • 493
  • 1
  • 6
  • 16
2
votes
0 answers

Messages not getting flushed to kafka

I have a topic with 5 partitions (A), and 5 consumers pulling messages from one partition each. Does some processing (takes about 30 seconds) and pushes to two topics with 1 partition each (Topic-B1, TopicB2). Topic-B1 is a log-compacted…
krishna reddy
  • 295
  • 2
  • 15
2
votes
2 answers

Write a csv file to a kafka topic

I have a large csv and I want to write to a kafka topic. def producer(): producer = KafkaProducer(bootstrap_servers='mykafka-broker') with open('/home/antonis/repos/testfile.csv') as file: reader = csv.DictReader(file,…
e7lT2P
  • 1,635
  • 5
  • 31
  • 57
2
votes
1 answer

Unable to register schema using register() in python

I am trying to register schema to confluent schema registry using python. from schema_registry.client import SchemaRegistryClient subject_name = "new-schema" schema_url = "https://{{ schemaRegistry }}:8081" sr =…
1689
  • 117
  • 1
  • 10
2
votes
1 answer

Suboptimal partitioning in Kafka consumer group

I have encountered a peculiar problem working with Kafka consumers. When I have a topic with a number of partitions, and a consumer group, the consumption eventually becomes unbalanced if consumer number is less than partition number. For example,…
StasM
  • 10,593
  • 6
  • 56
  • 103
2
votes
1 answer

Is there a way to create log compacted topic using kafka-python library?

kafka-python contains multiple modules to create/delete topic and also pass multiple configuration while doing so. Is there a way to add additional configuration to following method - NewTopic(name=topicname, num_partitions=1, replication_factor=1)
krishna reddy
  • 295
  • 2
  • 15
1 2 3
…
29 30