How to compute edges between nodes v, w that are pointed to by the same node x

Question

This question is about Spark GraphX. Given an arbitry graph, I want to compute a new graph that adds edges between any two nodes v, w that are both pointed to by some node x. The new edges should contain the pointing node as an attribute.

That is, given edges (x, v, nil) and (x, w, nil) compute edges (v, w, x) and (w, v, x).

It should work for any graph and not require me to know anything about the graph before hand, such as vertex ids.

Example

[Task] Add two directioned edges between nodes (e.g. A, C) when pointed to by same node (e.g. B).

Input graph:

          ┌────┐
    ┌─────│ B  │──────┐
    │     └────┘      │
    v                 v
 ┌────┐            ┌────┐
 │ A  │            │ C  │
 └────┘            └────┘
    ^                 ^
    │     ┌────┐      │
    └─────│ D  │──────┘
          └────┘

Output graph (bi-directional edges = two directed edges):

          ┌────┐
    ┌─────│ B  │──────┐
    │     └────┘      │
    v                 v
 ┌────┐<───by B───>┌────┐
 │ A  │            │ C  │
 └────┘<───by D───>└────┘
    ^                 ^
    │     ┌────┐      │
    └─────│ D  │──────┘
          └────┘

How to elegantly write a GraphX query that returns the output graph?

Your double-arrowed edge in the output graph does not make sense. Edges have a src and a dest -- which is which in the output graph? — David Griffin, May 19 '15 at 18:00
@DavidGriffin: you should read that as two directed edges. I'm going to update the question a bit now. — Pimin Konstantin Kefaloukos, May 20 '15 at 09:47
Btw, I'm currently working on a Pregel version of the solution. Would be nice to get your feedback on it. — Pimin Konstantin Kefaloukos, May 20 '15 at 10:44
No offense but that's not very elegant! How generalized do you need it to be? Because you could just run `graph.edges.flatMap` and for every edge just create a new one based off of it. That's basically all you are doing. — David Griffin, May 20 '15 at 15:34
@DavidGriffin: Perhaps I did not ask the question clearly enough. I'm (obviously?) looking for a general solution, not a solution to that particular example graph... In other words, the code should not require me to know the vertex IDs. My bad for not writing that in the question. — Pimin Konstantin Kefaloukos, May 20 '15 at 19:35
What else about the question needs to be generalized? Is it always 4 vertices? If it can be more than 4 vertices, where do the new vertices go, and where do the new Edges go? Like I said in the other comment, maybe describe what you are trying to solve. — David Griffin, May 20 '15 at 21:02
The query should work for any graph, e.g. a graph of 1 million nodes. It should compute all edges that match the description in my question. — Pimin Konstantin Kefaloukos, May 21 '15 at 08:20

Pimin Konstantin Kefaloukos · Answer 1 · 2015-05-20T19:23:06.330

Here is a solution that uses pregel and aggregate messages

import org.apache.spark.graphx._
import org.apache.spark.rdd.RDD

// Step 0: Create an input graph.
val nodes =
  sc.parallelize(Array(
    (101L, "A"), (102L, "A"), (201L, "B"), (202L, "B")
  ))
val edges = 
  sc.parallelize(Array(
    Edge(201L, 101L, ("B-to-A", 0L)), Edge(201L, 102L, ("B-to-A", 0L)),
    Edge(202L, 101L, ("B-to-A", 0L)), Edge(202L, 102L, ("B-to-A", 0L))
  ))    
val graph = Graph(nodes, edges, "default")

// Step 1: Transform input graph before running pregel.
val initialGraph = graph.mapVertices((id, _) => Set[(VertexId,VertexId)]())

// Step 2: Send downstream vertex IDs (A's) to upstream vertices (B's)
val round1 = initialGraph.pregel(
  initialMsg=Set[(VertexId,VertexId)](), 
  maxIterations=1, 
  activeDirection=EdgeDirection.In) 
(
  (id, old, msg) => old.union(msg),
  triplet => Iterator((triplet.srcId, Set((triplet.dstId, triplet.srcId)))),
  (a,b) => a.union(b)
)

// Step 3: Send (gathered) IDs back to downstream vertices
val round2 = round1.aggregateMessages[Set[(VertexId,VertexId)]](
  triplet => {
    triplet.sendToDst(triplet.srcAttr)
  },
  (a, b) => a.union(b)
)

// Step 4: Transform vertices to edges
val newEdges = round2.flatMap {v => v._2.filter(w => w._1 != v._1).map(w => Edge(v._1, w._1, ("shares-with", w._2)))}

// Step 5: Create a new graph that contains new edges
val newGraph = Graph(graph.vertices, graph.edges.union(newEdges))

// Step 6: print graph to verify result
newGraph.triplets foreach println

This solutions uses three main steps to compute a graph with the new edges: 1) a round of pregel. 2) a round of aggregateMessages. 3) a round of mapping nodes to edges.

How to compute edges between nodes v, w that are pointed to by the same node x

1 Answers1