Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
3
votes
3 answers

AssertionError: Unassigned partition

Im trying to consume data from a topic by setting the offset but get assertion error - from kafka import KafkaConsumer consumer = KafkaConsumer('foobar1', bootstrap_servers=['localhost:9092']) print 'process started' print…
user1050619
  • 19,822
  • 85
  • 237
  • 413
3
votes
2 answers

Flask - Pulling the live stream kafka data - Integrating Kafka with Python Flask

This project is for real time search engine - log analysis performance. I have a live streaming data out from Spark processing to Kafka. Now with the Kafka output, I want to get the data from the Kafka using Flask.. and visualize it using Chartjs…
Soundarya Thiagarajan
  • 574
  • 2
  • 13
  • 31
3
votes
2 answers

How to create new topic with pykafka with partitions and replication?

I want to be able to programmatically create a topic in Kafka using pykafka. I know that accessing the TopicDict will automatically create a topic if one does not exist but I don't know how to control the number of partitions/replicas with that. …
Tim Martin
  • 2,419
  • 1
  • 21
  • 25
3
votes
1 answer

Consume Kafka continuously and update queue at specific intervals using multiprocessing

I am trying to continuously consume events from kafka. The same application also uses this consumed data, to perform some analysis and update a database in n-second intervals (assume n = 60 seconds). In the same application, if process1 = Kafka…
anonuser0428
  • 11,789
  • 22
  • 63
  • 86
3
votes
1 answer

How to use kafka.consumer.SimpleConsumer,seek()

The API doc is here:http://kafka-python.readthedocs.org/en/latest/apidoc/kafka.consumer.html But when I run the following code, the exception is %d format: a number is required, not NoneType client = KafkaClient("localhost:9092") consumer =…
BAE
  • 8,550
  • 22
  • 88
  • 171
3
votes
1 answer

python kafka for consuming does not work

I can write to kafka. However, consuming does not work kafka = KafkaClient(kafka_host_list) consumer = SimpleConsumer(kafka,'topic-test-development','topic-test-development') No handlers could be found for logger "kafka.conn" Traceback (most…
Tampa
  • 75,446
  • 119
  • 278
  • 425
2
votes
1 answer

Integration tests with Kafka fail in first read, but then work correctly

I have custom function forward_to_kafka(list: List) which sends my events to kafka even if there are problems, it's purpose to deliver all messages in my list. I have already tested it with JB Plugin Big Data Tools and it works correctly. But now I…
shameoff
  • 61
  • 5
2
votes
1 answer

Kafka consumer not consuming the data produced into topic

I have to create a kafka producer that generates a sequence of numbers from 1 to 300. Each of the message I wrote have to contain information about the topic, a key and the value which is a the binary value of the value to write. This is the code…
Isabel Lopez
  • 101
  • 6
2
votes
2 answers

How can I use keystore.jks and truststore.jks files for Kafka Python SSL Authentication after converting to .pem?

I have 2 certificate files, truststore.jks and keystore.jks. keystore.jks contains a full certificate chain for the kafka endpoint I'm using as well as a private key for my application. According to the group who gave me truststore.jks it contains…
bayne
  • 31
  • 1
  • 5
2
votes
0 answers

Kafka consumer script hangs when executed as a docker entrypoint

I am trying to connect a python kafka consumer to a kafka broker using the following script: import kafka from configuration import config if __name__ == "__main__": print('Start') consumer = kafka.KafkaConsumer( …
Charalamm
  • 1,547
  • 1
  • 11
  • 27
2
votes
2 answers

pip install confluent-kafka gives error in mac

When i tried pip install confluent-kafka got the following error #include ^~~~~~~~~~~~~~~~~~~~~~ 1 error generated. error: command '/usr/bin/gcc' failed with exit code 1 I'm using python version 3.9…
Muhammed Fayis
  • 215
  • 3
  • 9
2
votes
1 answer

Difference between connections.max.idle.ms and max.poll.interval.ms in Kafka configuration?

I also want to know that since heartbeat requests are sent in Apache Kakfa Consumer, does it also affect the connections.max.idle.ms? How do you handle errors in kafka-apache client (Producer and Consumer and best practices around them)? Thanks :]
king.reflex
  • 313
  • 4
  • 10
2
votes
0 answers

KafkaConsumer prevents other asyncio.Task from running properly

I am trying to create an async Python app that subscribes to a Kafka topic and prints out the received messages via the asyncio.Task named echo_kafka_msgs_task. At the same time, there is a second asyncio.Task named ping_task that prints out an…
Athena Wisdom
  • 6,101
  • 9
  • 36
  • 60
2
votes
2 answers

Setting up docker-compose ports and host names to work with a kafka-python container

I'm learning Kafka at the moment, and struggling to get my docker-compose configuration set up properly. What I' trying to do is run a broker based on the wurstmeister/kafka image, and then another container that runs a simple python script with…
John Kealy
  • 1,503
  • 1
  • 13
  • 32
2
votes
1 answer

libraries for lz4 compression codec not found - kafka python

Error using kafka python (kafka-python 1.4.7) Kafka Consumer throwing error kafka.errors.UnsupportedCodecError: UnsupportedCodecError: libraries for lz4 compression codec not found
Smalltalkguy
  • 319
  • 1
  • 4
  • 13
1 2 3
…
29 30