Execute multiple actions parallel/async in Spark Streaming

Question

is there a way to execute multiple actions async/parallel in spark streaming? Here is my code:

positions.foreachRDD(rdd -> {

           JavaRDD<A> pbv = rdd.map(p -> A.create(p));          
           javaFunctions(pbv).writerBuilder("poc", "table_a", mapToRow(A.class)).saveToCassandra();     

           JavaRDD<D> pbd = rdd.map(p -> D.create(p));          
           javaFunctions(pbd).writerBuilder("poc", "table_d", mapToRow(D.class)).saveToCassandra();             

           JavaRDD<L> pblv = rdd.map(p -> L.create(p));         
           javaFunctions(pblv).writerBuilder("poc", "table_l", mapToRow(L.class)).saveToCassandra();            

           JavaRDD<V> pbld = rdd.map(p -> V.create(p));         
           javaFunctions(pbld).writerBuilder("poc", "table_v", mapToRow(V.class)).saveToCassandra();

    });

I would like to do the saveToCassandra actions in parallel, is this possible with "spark technics" or only via selfmade Thread/Executer handling?

Thanks for your help!

Regards, Markus

Not exactly an answer to your question, but the fastest way to execute these inserts is to do `rdd.foreachPartition`, create `PreparedStatements` using the java driver and submit them with `session.executeAsync` — maasg, Jun 16 '16 at 13:53
Thx for the hint. Will try to test your approach. if the dstream approach from @RussS didn't work. — mananana, Jun 17 '16 at 06:53

score 1 · Accepted Answer · answered Jun 16 '16 at 19:02

1

@massg's comment is probably the fastest approach to this kind of forking but if you don't want that you could more easily do this by just splitting the stream.

Something like

positions.map(A.create(p)).saveToCassandra
positions.map(D.create(p)).saveToCassandra
positions.map(L.create(p)).saveToCassandra
positions.map(V.create(p)).saveToCassandra

By acting on the DSTream all these requests will happen in parallel.

answered Jun 16 '16 at 19:02

RussS

16,476
1
34
62

Changed my code to your approach: JavaDStream pbv = positions.map(p -> A.create(p)); CassandraStreamingJavaUtil.javaFunctions(pbv).writerBuilder("poc", "table_a", mapToRow(A.class)).saveToCassandra(); ... But it seems not like they are executeted in parallel now. Did I miss something? What I see now, is, that there are now 4 output operations, instead of on before. But the processing time seems like the sum of the 4 operations. (Expect, that the processing time should be not the sum, but only the time of the longest operation) – mananana Jun 17 '16 at 06:50
Hmm those jobs should all have the same dependency so they should be executed at the same time, of course one operator can still take up all of the computational resources if you don't have enough cores. – RussS Jun 17 '16 at 17:07
If you use the FIFO scheduler which is the default it will execute the outputs one by one. If you even cache the positions RDD before saving (maybe because the computation of the RDD is complex) it should be ok not to execute them in parallel as the first output computes the output and the remaining 3 outputs just store the saved cache – mniehoff Jun 20 '16 at 09:00

Execute multiple actions parallel/async in Spark Streaming

1 Answers1