1

I am trying to write a spark evaluator in Streamsets. I have to deal with complex SQL queries and hence would want to use data frames or datasets here. But the sample code which Streamsets provides deals with JavaRDD only. Can I have an insight on dataframe to get some headstart here ?

earl
  • 738
  • 1
  • 17
  • 38

1 Answers1

0

You are almost certainly better off looking at using StreamSets Transformer. Transformer has a much deeper Spark integration and will allow you to work with native Spark structures.

metadaddy
  • 4,234
  • 1
  • 22
  • 46