0

I trying to process the edges of a map. Each edge represents a path between two nodes as shown in the figure

links

From the map data I have, I am able to get this kind of data structure.

output: ArrayBuffer((1,4), (1,2), (2,4), (2,1), (2,3), (2,5), (3,2), (3,5), (4,1), (4,2), (5,2), (5,3))

The goal is to join connected segments/paths to calculate the combined length. The order of joining doesn't matter here as I only need to find the combined length after joining, to check if it satisfy a given condition. The possible joining segments could be:

>(1,2,3) [or even 3,2,1 as order doesn't matter]
>(1,2,5) or (5,2,1)
>(4,2,3) or (3,2,4)
>(4,2,5) or (5,2,4)

The problem here is: I should not consider paths like (1,2,4), (1,4), (3,2,5) or (3,5) as they are not feasible paths. I am trying to implement this in Spark using Scala as real data is huge map data. As I am new to this domain, any help would be greatly appreciated. P.S: I have the source from where I can get the length of each path/segment to add those lengths together after knowing which segments should be joined together.

UPDATE

processing each node for a given link gives me this: output: 

ArrayBuffer((1,List()), (1,List(4, 2)), (2,List(4, 1)), (2,List(3, 5)), (3,List(2, 5)), (3,List()), (4,List(1, 2)), (4,List()), (5,List(2, 3)), (5,List()) (1,List())

because left node of this segment is not connected to any other segment and we can actually ignore all these single segments from our resulting structure. I got this final structure using scala map and flatmap function output: 

ArrayBuffer((1,4), (1,2), (2,4), (2,1), (2,3), (2,5), (3,2), (3,5), (4,1), (4,2), (5,2), (5,3))
Vladislav Varslavans
  • 2,775
  • 4
  • 18
  • 33
Waleed
  • 66
  • 3
  • UPDATE: processing each node for a given link gives me this: output: ArrayBuffer((1,List()), (1,List(4, 2)), (2,List(4, 1)), (2,List(3, 5)), (3,List(2, 5)), (3,List()), (4,List(1, 2)), (4,List()), (5,List(2, 3)), (5,List()) **(1,List()) because left node of this segment is not connected to any other segment** and we can actually ignore all these single segments from our resulting structure. I got this final structure using scala map and flatmap function output: ArrayBuffer((1,4), (1,2), (2,4), (2,1), (2,3), (2,5), (3,2), (3,5), (4,1), (4,2), (5,2), (5,3)) – Waleed Dec 09 '19 at 11:55

1 Answers1

0

Consider using GraphX from spark. You could make each edge from your data to be an edge in the GraphX. And then calculate paths between vertices you need.

UPDATE

When you create nodes, like A, B, C, D, E, F on the image below - you will find that some of them only have one edge. You can use that condition to find possible start and end nodes of your path. Besides that you need to manually specify which nodes can or cannot be start or end of your paths. Maybe there is some another information that could be used for that. enter image description here

Vladislav Varslavans
  • 2,775
  • 4
  • 18
  • 33
  • Thanks for your answer but I am still not able to understand how, in any ways, i will be able to handle the condition where (1,2,4), (1,4), (3,2,5) or (3,5) should not be considered while joining segments. – Waleed Dec 09 '19 at 13:41
  • You could also update question with explanation why path `A->C->B` (1, 4) is not a valid path – Vladislav Varslavans Dec 09 '19 at 14:00