Vertex Property Inheritance - Graphx Scala Spark

Question

--- Edit ---

My main issue is that I do not understand this paragraph given in the Graphx documentation:

In some cases it may be desirable to have vertices with different property types in the same graph. This can be accomplished through inheritance. For example to model users and products as a bipartite graph we might do the following:

class VertexProperty()
case class UserProperty(val name: String) extends VertexProperty
case class ProductProperty(val name: String, val price: Double) extends VertexProperty
// The graph might then have the type:
var graph: Graph[VertexProperty, String] = null

In the above case given RDD's of each UserProperty and ProductProperty and a RDD of EdgeProperty, how does one create a graph of type Graph[VertexProperty, String]. I am looking for an example.

score 2 · Answer 1 · answered Apr 19 '15 at 23:08

This will help you to create a bipartite graph, where vertex property will help you to understand the different class categories.

// High level interface OR VertexProperty

trait Node {   def getVertexID : Long  }

class UserNode(sID: String, sname : String, sAge) extends Node with Serializable { }

class ProductNode(sID: String, sNO : String, sdoe : String) extends Node with Serializable{ }

// Data loading

val users: RDD[Node]  = sc.textFile("users.txt")
                                 .map { row =>  val cols = row.split(",")
                                         ( new UserNode(cols(0), cols(1), cols(2))
                                  }

val products: RDD[Node]  = sc.textFile("products.txt")
                                 .map { row =>  val cols = row.split(",")
                                        ( new ProductNode(cols(0), cols(1), cols(3)))
                                }

// Join both RDDs

 val nodes : RDD[Node] = users.++(products)

score 0 · Answer 2 · answered Mar 10 '15 at 14:46

0

You can use a message which can be merged, for example an Iterable[YourClass]. However you have to take into consideration that the size of these kind of merges can become very large.

answered Mar 10 '15 at 14:46

Sietse

201
1
4

score 0 · Accepted Answer · answered Mar 11 '15 at 20:39

It is a scala question, just convert the the extended type to the abstract type using asInstanceOf, for example:

val variable1: RDD[UserProperty]  = {..your code..}
val variable2: RDD[ProductProperty]  = {..your code..}
val result: RDD[VertexProperty] = SparkContext.union(
variable1.asInstanceOf[VertexProperty],
variable2.asInstanceOf[VertexProperty])

The same goes for edge property, use

val edge: EdgeProperty = Edge(srcID, dstID, variable.asInstanceOf(EdgeProperty))

Vertex Property Inheritance - Graphx Scala Spark

3 Answers3