How can I receive messages as RDD instead of DStream from a realtime source with a custom receiver?

Asked Jul 11 '19 at 06:37

Active Jul 11 '19 at 06:37

Viewed 42 times

I want to get RDD from a realtime message source (currently socket) to replace the line of code that is used to read from a local file sc.newAPIHadoopFile(). In order to maintain the structure of the project, I have to get RDD instead of DStream from that source. I tried to implement a custom receiver, but StreamingContext.receiverStream() only gives me a DStream. How can I do?

I tried to convert a DStream to RDD with compute, but that doesn’t make any sense and I failed. So I turned to discover how to directly receive RDD with a receiver.

asked Jul 11 '19 at 06:37

William Wang

`DStream.foreachRDD` – ollik1 Jul 11 '19 at 06:46
Do you mean union all the RDD’s from an DStream? Can you provide more details plz? thanks a lot. – William Wang Jul 11 '19 at 06:56

How can I receive messages as RDD instead of DStream from a realtime source with a custom receiver?

0 Answers0