0

I'm currently using graphframes to generate a graph and then I need to find the paths between all vertices. (That is all pairs of vertices are tested to find the minimum path between them.)

Both bfs and find in graphframes generate dataframes meaning they cannot be used in any kind of mapping function. Is there a way I can work around this? (For instance, doing a for loop on all RDD entries in a partition [mapPartitions doesn't work, I tried it.])

Dylan Lawrence
  • 1,503
  • 10
  • 32
  • It might be possibble to use bfs or find, and just map the output of that back to an RDD before returning. Would that work? – Davis Broda Feb 27 '17 at 21:53
  • @DavisBroda I tried that, it seems Spark doesn't care, if at any point inside the map you create a dataframe or RDD it throws an error. – Dylan Lawrence Feb 27 '17 at 22:08

0 Answers0