0

Is is possible to get multiple DStream out of a single DStream in spark. My use case is follows: I am getting Stream of log data from HDFS file. The log line contains an id (id=xyz). I need to process log line differently based on the id. So I was trying to different Dstream for each id from input Dstream. I couldnt find anything related in documentation. Does anyone know how this can be achieved in Spark or point to any link for this.

Thanks

Alok
  • 1,374
  • 3
  • 18
  • 44

1 Answers1

0

You cannot Split multiple DStreams from Single DStreams. The best you can do is: -

  1. Modify your source system to have different streams for different ID's and then you can have different jobs to process different Streams
  2. In case your source cannot change and provide you stream which is mix of ID, then you need to write custom logic to identify the ID and then perform the appropriate operation.

I would always prefer #1 as that is cleaner solution but there are exceptions for which #2 needs to be implemented.

Sumit
  • 1,400
  • 7
  • 9
  • Thanks . So i Believe I also cannot create multiple dataframe from source based on id ? – Alok Jan 20 '16 at 10:48
  • The way I would do it is - first create DataFrame and then filtering DataFrame on some criteria to create further filtered dataframes or perform some action on filtered data. – Sumit Jan 20 '16 at 11:08