New developer in Scala here and also a new user of Spark GraphX. So far, I'm really enjoying my time but I've just had a really strange bug. I have isolated the problem to a long-to-int conversion but it's really weird. Another weird thing is that it works fine in Windows, but doesn't work in Linux (creates an infinite loop) I have located the source of the problem in Linux but I don't understand why there is a problem. I have to put the random number inside a variable first and then it works.
You should be able to copy/paste and execute the whole thing
Scala 2.10.6, Spark 2.1.0, Linux Ubuntu 16.04
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.graphx._
import scala.util.Random
object Main extends App {
//Fonction template pour imprimer n'importe quel graphe
def printGraph[VD,ED] ( g : Graph[VD,ED] ): Unit = {
g.vertices.collect.foreach( println )
}
def randomNumber(limit : Int) = {
val start = 1
val end = limit
val rnd = new Random
start + rnd.nextInt( (end - start) + 1 )
}
val conf = new SparkConf()
.setAppName("Simple Application")
.setMaster("local[*]")
val sc = new SparkContext(conf)
sc.setLogLevel("ERROR")
val myVertices = sc.makeRDD(Array((1L, "A"), (2L, "B"), (3L, "C"), (4L, "D"), (5L, "E"), (6L, "F")))
val myEdges = sc.makeRDD(Array(Edge(1L, 2L, ""),
Edge(1L, 3L, ""), Edge(1L, 6L, ""), Edge(2L, 3L, ""),
Edge(2L, 4L, ""), Edge(2L, 5L, ""), Edge(3L, 5L, ""),
Edge(4L, 6L, ""), Edge(5L, 6L, "")))
val myGraph = Graph(myVertices, myEdges)
//Add a random color to each vertice. This random color is chosen from the total number of vertices
//Transform vertex attribute to color only
val bug = myVertices.count()
println("Long : " + bug)
val bugInt = bug.toInt
println("Int : " + bugInt)
//Problem is here when adding myGraph.vertices.count().toInt inside randomNumber. Works on Windows, infinite loop on Linux.
val g2 = myGraph.mapVertices( ( id, name ) => ( randomNumber(myGraph.vertices.count().toInt) ))
//Rest of code removed
}
