Combine SpqrkSQL and GraphX

Question

Can you create a stored procedure in SparkSQL and call GraphX API? something like this:

registerFunction("storedProcedureGraphX", model.storedProcedureGraphX _)

select * from someTable where storedProcedureGraphX(nodeX, nodeY) > 10

score 0 · Accepted Answer · answered Jul 21 '15 at 22:29

If by GraphX API you mean any operation on RDD then the answer is no. That would involve launching new Spark task for each row which definitely isn't a good idea. And you'd have to close on the SparkContext, which is not serializable - and functions that you use as UDFs have to be.

Combine SpqrkSQL and GraphX

1 Answers1