spark stream kafka parallel receiver receive data unbalanced

Asked Dec 08 '16 at 10:10

Active Dec 08 '16 at 11:39

Viewed 76 times

I just want try to receive stream data from kafka in parallel. Here is my code:

val myKafkaStream = (1 to numReceivers.toInt).map { i =>
   KafkaUtils.createStream(ssc, zkQuorum, group, topicMap)
}

I run the code on the yarn , the numReceiver is 5,and the code does work. But the question is only two receivers receive data. the num-executor is 5,the executor-core is 2

The result:

KafkaReveiver-0 : 49637.95 event/sec
KafkaReveiver-1 : 0.0 event/sec
KafkaReveiver-2 : 0.0 event/sec
KafkaReveiver-3 : 49616.85 event/sec
KafkaReveiver-4 : 0.0 event/sec

edited Dec 08 '16 at 11:39

Yuval Itzchakov

146,575
32
257
321

asked Dec 08 '16 at 10:10

Stone

Please can you post what you have assigned to topicMap – ImDarrenG Dec 08 '16 at 10:20
`val topicMap = topics.split(",").map((_,60)).toMap` and the topic is "astro_spark" – Stone Dec 08 '16 at 11:02

spark stream kafka parallel receiver receive data unbalanced

0 Answers0