Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
4
votes
2 answers

kafka-python 1.3.3: KafkaProducer.send with explicit key fails to send message to broker

(Possibly a duplicate of Can't send a keyedMessage to brokers with partitioner.class=kafka.producer.DefaultPartitioner, although the OP of that question didn't mention kafka-python. And anyway, it never got an answer.) I have a Python program that…
Hephaestus
  • 1,982
  • 3
  • 27
  • 35
4
votes
1 answer

kafka consumer seek is not working: AssertionError: Unassigned partition

the kafka consumer con defined below works perfectly fine when I try to receive messages form my topic; however, it's giving me trouble when I try to change the offset using seek method or any of its variations. i.e. seek_to_beginning,…
Zahra
  • 6,798
  • 9
  • 51
  • 76
4
votes
2 answers

Encoding / formatting issues with python kafka library

I've been trying to use the python kafka library for a bit now and can't get a producer to work. After a bit of research I've found out that kafka sends (and I'm guessing expects as well) an additional 5 byte header (one 0 byte, one long containing…
Moises Jimenez
  • 1,962
  • 3
  • 21
  • 43
4
votes
1 answer

Create a Kafka brokers cluster using Apache Kafka 0.10.0 API with Java

I want to create a broker cluster using Kafka 0.10 API preferable with Java. As far as I have read kafka_2.11-0.10.0.0.jar do support creating broker using : import kafka.cluster.Broker; import kafka.cluster.Cluster; But I can't find any…
gourxb
  • 325
  • 1
  • 5
  • 13
4
votes
2 answers

How to use kafka on tornado?

I'm trying to make a simple chat app using tornado based on this But also I want to use kafka to store the messages. How can I do that? Now, I used this to make a consumer and somehow it's working but it's only printing on the console and I need…
johnjullies
  • 745
  • 1
  • 8
  • 22
4
votes
2 answers

Python how to delete all messages under a Kafka topic

I am new to kafka. We are trying to import data from a csv file to Kafka. We need to import everyday, in the mean while the previous day's data is depredated. How could remove all messages under a Kafka topic in python? or how could I remove the…
lucky_start_izumi
  • 2,511
  • 13
  • 41
  • 61
3
votes
1 answer

Kafka + FastAPI + Docker template

Introduction I am currently experimenting with Kafka and FastAPI and trying to build a template to enable me to quickly write programs in a microservice pattern. Goal - Vision Building a repository of design patterns that implement very easy…
mm117
  • 81
  • 6
3
votes
1 answer

Kafka-python Producer performs Send, but no data arrives in Kafka

I am attempting to send data to a Kafka topic via python as a remote producer. My script reports no exceptions, but nothing shows up in Kafka. I'm able to get a response from the brokers (all on one separate host) using the consumer method. Looking…
Dav25557
  • 31
  • 4
3
votes
1 answer

Kafka One particular consumers group is rebalancing

I created few month ago two consumers groups (each group has 3 consumers) in my Kafka chain : First group : take data from external source and send them to my topic1 Second group : consume from topic1 This without problems. Recently, I added a…
Chris
  • 234
  • 3
  • 16
3
votes
1 answer

How to programmatically update subject schema and compatibility in Confluent Schema Registry

I have a schema already registered in schema registry, which I was able to do using register() like this, from schema_registry.client import SchemaRegistryClient, schema subject_name = "new-schema" schema_url = "https://{{ schemaRegistry }}:8081"…
1689
  • 117
  • 1
  • 10
3
votes
2 answers

Mocking python kafka producer

I have a file with a function which creates a Kafka Producer and publishes some message on to Kafka topic. def publishToKafkaTopic(value_str): producer = KafkaProducer( bootstrap_servers=KAFKA_SERVERS, value_serializer=lambda x:…
ace
  • 139
  • 2
  • 6
3
votes
1 answer

How to programatically register Avro Schema in Kafka Schema Registry using Python

I put data and schema to kafka and schema registry with python. from confluent_kafka import avro from confluent_kafka.avro import AvroProducer value_schema_str = """ { "type":"record", "name":"myrecord", "fields":[ …
CompEng
  • 7,161
  • 16
  • 68
  • 122
3
votes
2 answers

How to produce a Tombstone Avro Record in Kafka using Python?

my sink properties : { "name": "jdbc-oracle", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max": "1", "topics": "orders", "connection.url": "jdbc:oracle:thin:@10.1.2.3:1071/orac", …
CompEng
  • 7,161
  • 16
  • 68
  • 122
3
votes
0 answers

kafka-python SASL/SCRAM-SHA-256 auth

I need to connect to Kafka broker in python using SASL/SCRAM-SHA-256 authentication. I use the kafka-python (2.0.1) python library which allows this kind of connection/authentication. This is an example of code, but i get the error ... not Brokers…
Alfonso
  • 113
  • 9
3
votes
0 answers

Produce/Consume to Remote Kafka Does not Work

I have set up a AWS EC2 instance running Apache Kafka 0.8 via a Bitnami AMI image. The server properties are pretty much default (Kafka located at localhost:9092 and zookeeper located at localhost:2181). When I SSH into the machine, I can…
exactapproximate
  • 31
  • 3
1 2 3
…
29 30