Spark Streaming Run Actions On DStream Asynchronously

Question

I'm writing a program for data ingestion. Read from Kafka to DStream split the Dstrem to 3 streams and executing Actions on each one:

val stream = createSparkStream(Globals.configs, ssc)
val s1 = stream.filter(<predicat1>)
val s2 = stream.filter(<predicat2>)
val s3 = stream.filter(<predicat3>)

//I'm looking for something like:
s1.forEachRddAsync(...
s2.forEachRddAsync(...
s3.forEachRddAsync(...

If it possible to trigger async submit on whole DStream and not RDD.

score 0 · Answer 1 · answered Oct 14 '18 at 09:33

0

DStream action methods, while indeed blocking, don't process data. These only register DStream as an output stream.

Once the StreamingContext is started, the processing will be scheduled according to the available resources, and if these allow, be processed without limiting each other.

answered Oct 14 '18 at 09:33

user10502122

1

From what I see in spark UI, all foreachRdd actions run sequentially – Alex Oct 14 '18 at 16:01

Spark Streaming Run Actions On DStream Asynchronously

1 Answers1