How do I make the join between two streams on Spark structured streaming? - Java 8

Question

I know, join between two streams is not supported yet. But there's a way to do it anyway? Or I need to wait for Spark 2.2.0? Thank you

Did you find a way to do this with structured streaming? – fledgling Dec 01 '17 at 20:41 — fledgling, Dec 01 '17 at 20:41

score 4 · Accepted Answer · answered Apr 20 '17 at 18:16

4

Spark 2.2.0 still won't support stream-stream join. You can watch this ticket: https://issues.apache.org/jira/browse/SPARK-18791

answered Apr 20 '17 at 18:16

zsxwing

will it be available in 2.3 ? – user1870400 Feb 12 '18 at 04:10
1

Yes. 2.3 will support it. – zsxwing Feb 12 '18 at 08:34
Hi! I was just playing with spark 2.3 RC4 stream-stream join. Somehow I couldn't do a self join on streaming dataframe easily. and I am not sure why..It might be my syntax but I am not sure..more details here https://stackoverflow.com/questions/48900127/how-to-do-a-self-join-in-spark-2-3-0-what-is-the-correct-syntax would be great if you can help on this one!! – user1870400 Feb 21 '18 at 08:07

score 1 · Answer 2 · answered Mar 04 '18 at 22:20

1

As of production release of Spark 2.3, stream-stream joins are now supported in structured streaming.

answered Mar 04 '18 at 22:20

rhoeting

score -2 · Answer 3 · answered May 19 '17 at 10:50

-2

I recommend you using Kafka Streams rather than Structured Spark Streaming if you want to join streams.. check this link to see how this can be done

answered May 19 '17 at 10:50

Gilles Essoki

Thank you, but I've already used Kafka Streams. I want to try some different streaming engines for specific reason :D – Aniello Guarino May 19 '17 at 12:48

3 Answers3