2

I'm trying to run the connected component algorithm on my dataset but on a directional graph. I don't want the connected component to transverse in both direction of the edges.

This is my sample code 

import org.apache.log4j.{Level, LogManager}
import org.apache.spark.SparkConf
import org.apache.spark.graphx.Edge
import org.apache.spark.sql._
import org.graphframes._

object CCTest {

def main(args: Array[String]) {

  val sparkConf = new SparkConf()
      .setMaster("local[2]")
    .setAppName("cc_test")


  implicit val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate()

  val sc = sparkSession.sparkContext

  val vertex = sparkSession.createDataFrame(sc.parallelize(Array(
    (1L, "b4fcde907cbd290b7e16", 28),
    (2L, "cda389612d6b37674cb1", 27),
    (3L, "1a6a6e3fd2daaeeb2a05", 65),
    (4L, "9a007eee210a47e58047", 42),
    (5L, "e91898d39bf5f8501827", 55),
    (6L, "ceab58c59d23549d3f4b", 50),
    (12L, "ceab58c59asd3549d3f4b", 50),
    (14L, "ceab508c59d23549d3f4b", 55),
    (15L, "ceab508c59d23541d3f4b", 51)
  ))).toDF("id", "similar_hash", "size")

  val edges = sparkSession.createDataFrame(sc.parallelize(Array(
            Edge(2L, 1L, 0.7f),
            Edge(2L, 4L, 0.2f),
            Edge(3L, 2L, 0.4f),
            Edge(3L, 6L, 0.3f),
            Edge(4L, 1L, 0.1f),
            Edge(5L, 2L, 0.2f),
            Edge(5L, 3L, 0.8f),
            Edge(5L, 6L, 0.3f),
            Edge(12L, 14L, 1.3f),
            Edge(15L, 14L, 1.3f)    //< - should not be connected except (14L, 15L)
          ))).toDF("src", "dst", "attr")


  val graph = GraphFrame(vertex, edges)

  val cc = graph.connectedComponents.run()

  cc.show()



  sparkSession.stop()

}

 }

Result:

+---+--------------------+----+---------+
| id|        similar_hash|size|component|
+---+--------------------+----+---------+
|  6|ceab58c59d23549d3f4b|  50|        1|
|  5|e91898d39bf5f8501827|  55|        1|
|  1|b4fcde907cbd290b7e16|  28|        1|
|  3|1a6a6e3fd2daaeeb2a05|  65|        1|
| 12|ceab58c59asd3549d...|  50|       12|
|  2|cda389612d6b37674cb1|  27|        1|
|  4|9a007eee210a47e58047|  42|        1|
| 14|ceab508c59d23549d...|  55|       12|
| 15|ceab508c59d23541d...|  51|       12|   <- should be in separate cluster
+---+--------------------+----+---------+

How can I achieve this please?

Philip K. Adetiloye
  • 3,102
  • 4
  • 37
  • 63
  • 2
    [`stronglyConnectedComponents`](https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.graphx.GraphOps@stronglyConnectedComponents(numIter:Int):org.apache.spark.graphx.Graph[org.apache.spark.graphx.VertexId,ED]). – zero323 Feb 27 '17 at 22:50
  • @zero323 I don't think SCC can be used here (correct me if I'm wrong) since it requires every node to be reachable from other nodes. But basically, I need to do a recursion like SQL recursive query. For example: a -> b, b -> c, m -> b Strongly CC: Set = {a, b}, Set = {b, c}, Set = {m, b} Connected component: It will be the set = `{a, b, d, m}` -- notice and `m` connected to the node`a -> b -> m` too But in recursion a -> b -> c = {a, b, c} and m -> b = {m, b} this give the two sets {a, b, c} and {m, b} – Philip K. Adetiloye Feb 28 '17 at 11:18
  • OK. So why `{a, b, c}`, `{m, b}` and not `{a, b, c}`, `{m, b, c}`? One way or another it looks like you'll need Pregel API for that. – zero323 Feb 28 '17 at 12:41
  • @zero323 correct - {a, b, c} and {m, b, c}. I thought about the Pregel API. I'll give that a try too - Thank you – Philip K. Adetiloye Feb 28 '17 at 13:02
  • If I may suggest I would try to clarify the problem. It look like you're looking more for DFS or reachability graph than connected components. – zero323 Feb 28 '17 at 13:36

1 Answers1

1

I might be mistaken, but my solution stems directly from the source code of connectedComponents. In line 54, the system itself only calls a Pregel iterator, with the key line being

val pregelGraph = Pregel(ccGraph, initialMsg, maxIterations, EdgeDirection.Either)

By simply changing the EdgeDirection to a more suitable parameter (can be found here), this could work for you.

dennlinger
  • 9,890
  • 1
  • 42
  • 63