Stack Overflow
Stack Overflow

Questions tagged [kafka-python]

Kafka-Python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

kafka-python provides low-level protocol support for Apache Kafka as well as high-level consumer and producer classes. Request batching is supported by the protocol as well as broker-aware request routing. Gzip and Snappy compression is also supported for message sets.

For more details about Python Kafka Client API, please refer https://kafka-python.readthedocs.io/en/latest/

443 questions
0
votes
1 answer

Kafka producer using python: TypeError: all produce message payloads must be null or type bytes

I just started learning Python and Kafka. This is the first example I tried to get started. http://www.giantflyingsaucer.com/blog/?p=5541 And I got an exception: Traceback (most recent call last): File "producer.py", line 23, in …
gogocatmario
  • 70
  • 10
0
votes
0 answers

Producer can not connect to broker through DNS

I have a physical server, where i have set the advertised.host.name to server ip, and do the port forwarding at the router. But the producer cannot connect to broker by using the dns. ERROR:pykafka.connection:Failed to connect to…
Yong
  • 1
  • 4
0
votes
1 answer

Single producer to multi consumers (Same consumer group)

I've try before sending message from single producer to 2 different consumer with DIFFERENT consumer group id. The result is both consumer able to read the complete message (both consumers getting the same message). But I would like to ask is it…
不好笑
  • 105
  • 3
  • 14
0
votes
3 answers

Kafka Consumer didn't receiving any messages from its Producer

the following is my python coding for a kafka producer, I'm not sure is the messages able to be published to a Kafka Broker or not. Because the consumer side is didn't receiving any messages. My Consumer python program is working fine while i…
不好笑
  • 105
  • 3
  • 14
0
votes
1 answer

Why is there a delay between writing to and reading from Kafka queue?

I have written a worker service to consume messages from a Kafka queue, and I have also written a test script to add messages to the queue every few seconds. What I have noticed is that often the consumer will sit by idle for minutes at a time,…
Ben Harrison
  • 2,121
  • 4
  • 24
  • 40
0
votes
1 answer

python kafka: how to make each msg consumed only once by group from begining

I am using Kafka consumer here (version 1.3.1). What I am going to acheive: There are 10 partitions. each partition begins with offset 0. There is a group of consumers (1,2,3, eg). Sometimes, one consumer is down or is up. So, the group…
BAE
  • 8,550
  • 22
  • 88
  • 171
0
votes
1 answer

How to combine two DStreams(pyspark)?

I have a kafka stream coming in with some input topic. This is the code i wrote for accepting kafka stream. conf = SparkConf().setAppName(appname) sc = SparkContext(conf=conf) ssc = StreamingContext(sc) kvs = KafkaUtils.createDirectStream(ssc,…
vidhan
  • 129
  • 2
  • 10
0
votes
1 answer

pyspark is unable to find KafkaUtils.createDirectStream

I have the following pyspark script, which suppose to connect to a local kafka cluster: from pyspark import SparkConf, SparkContext from operator import add import sys from pyspark.streaming import StreamingContext from pyspark.streaming.kafka…
Eugene Goldberg
  • 14,286
  • 20
  • 94
  • 167
0
votes
1 answer

Error message with Kafka basic setup creating topic and sending messages

I am trying to setup my Kafka but I have some key issues. I can't even create a topic or send messages. My Kafka Version number is: 0.9.0.1 Thanks for your help. Below please find my error message: [2016-04-21 04:41:19,306] producer ERROR…
peter
  • 674
  • 1
  • 12
  • 33
0
votes
1 answer

Kafka : Does Kafka provide support for application level state transition?

Suppose, I have one topic with 6 partitions and 2 consumers where P1, P2, P3 processed by C1 and P4, P5, P6 processed by C2. Let us say user data U1 always goes to P1, U2 to P2 and so on. So, C1 maintains state of users U1, U2, U3 C2 maintains…
user3851499
0
votes
1 answer

Mulitple Python Consumer Threads on a Single Partition with Kafka 0.9.0

For context, I am trying to transfer our python worker processes over to a kafka (0.9.0) based architecture, but I am confused about the limitations of partitions with respect to the consumer threads. Will having multiple consumers on a partition…
M. Carlson
  • 788
  • 7
  • 17
0
votes
2 answers

How to differentiate topics received in kafka using spark streaming

I am using the following code to get the message from the kafka scala code: val lines: ReceiverInputDStream[(String, String)] = KafkaUtils.createStream(ssc, zookeeperQuorum, consumerGroup, topicMap) lines.print(10) Here is my sample producer code…
Knight71
  • 2,927
  • 5
  • 37
  • 63
0
votes
1 answer

How can I improve the following code performance to ingest 1 million record /second

The following code are ingesting 10k-20k record per second and I want to improve the performance of it. I am reading a json format and ingesting it into database using Kafka. -I am running it on the cluster of five nodes with zookeeper and Kafka…
Ahmed Alashrafy
  • 83
  • 6
0
votes
1 answer

Using Kafka low-level API, should I commit the offset when finished fetching data?

public void run() { // find the meta data about the topic and partition we are interested in PartitionMetadata metadata = findLeader(a_seedBrokers, a_port, a_topic, a_partition); if (metadata == null) { System.out.println("Can't…
Sinme
  • 1
  • 2
0
votes
1 answer

Pushing the PubMed Data to Kafka

In the PubMed Data Source, I need to push the Output into a Kafka queue..Each source could be viewed as a Kafka Topic. (I know the concepts in Kafka and explored Kafka using Python) I am able to view the PubMed Data(s) through FireFTP. Can anyone…
Soundarya Thiagarajan
  • 574
  • 2
  • 13
  • 31
1 2 3
…
29
30