Stack Overflow
Stack Overflow

Questions tagged [spark-graphx]

GraphX is a component in Apache Spark for graphs and graph-parallel computation

GraphX is a component in Apache Spark for graphs and graph-parallel computation.

At a high level, GraphX extends the Spark RDD by introducing a new Graph abstraction: a directed multigraph with properties attached to each vertex and edge.

To support graph computation, GraphX exposes a set of fundamental operators (e.g., subgraph, joinVertices, and aggregateMessages) as well as an optimized variant of the Pregel API.

In addition, GraphX includes a growing collection of graph algorithms and builders to simplify graph analytics tasks.

487 questions
0
votes
1 answer

Scala - Spark : return vertex properties from particular node

I have a Graph and I want to compute the max degree. In particular the vertex with max degree I want to know all properties. This is the snippets of code: def max(a: (VertexId, Int), b: (VertexId, Int)): (VertexId, Int) = { if (a._2 > b._2) a…
alukard990
  • 811
  • 2
  • 9
  • 14
0
votes
1 answer

scala type mismatch error in graphX code

I'm new to scala, learning it for apache-spark. I wrote a simple function in scala for graphX def foo(edge: EdgeTriplet[Map[Long, Double], Double]): Iterator[(VertexId, Map[Long, Double])] = { val m = edge.srcAttr for((k, v) <- m){ …
pauli
  • 4,191
  • 2
  • 25
  • 41
0
votes
1 answer

Spark - Simple GraphX program taking long time to complete

I have a 9 node m3.xlarge (8 cpu / 15 gig) EMR cluster, where 1 node is the master and other 8 are slaves. I'm trying to run a simple program to check GraphX connected components. This is my code: def main(args : Array[String]): Unit = { val…
drunkenfist
  • 2,958
  • 12
  • 39
  • 73
0
votes
1 answer

GraphX pregel and spark streaming: the RDDs pushed into the rddQueue within the vprog are not processed

I am using the GraphX pregel and the spark streaming. I want that the vertex program (the vprog) creates a RDD and pushes it into the rddQueue to be processed. val queueOfRDDs:Queue[RDD[Int]] = Queue.empty[RDD[Int]] @transient val…
DaliMidou
  • 111
  • 1
  • 3
  • 14
0
votes
1 answer

Collecting neighboring vertices graphx

I'm trying to collect all the neighboring vertices given one vertex. But I am getting this error. An example of one vertex is (1,"some string") val direction: EdgeDirection = graph.collectNeighborIds(direction).lookup(1) Here is my…
squad21
  • 73
  • 2
  • 8
0
votes
1 answer

SparkException: Job aborted due to stage failure: NullPointerException when working with Spark-Graphx

I'm new in scala and I'm looking for solving this error. The scenario I'm working on is this. I've 3 tables: user: containing ID and name business: containing ID and name reviews: containing user.ID and business.ID Only users make a review and…
TheEnigmist
  • 906
  • 1
  • 10
  • 23
0
votes
1 answer

Query regarding Canonical Orientation in Graphx Triangle Count

I have read many articles along with Spark own documentation which says Graphx Triangle Count requires Graph to be Canonically Orientated. About canonical orientation, it is described as for an edge source_id < dest_id. With following three edges…
Sourav Gulati
  • 1,359
  • 9
  • 18
0
votes
1 answer

Spark-graphx- Strongly connected components

I'm new to spark and spark-graphx. I tried to run strongly connected components, but I'm getting only the triplets which are connected. What I'm looking for here is getting all the vertices (group of vertices), which are only strongly connected…
spark-graphx novice
  • 25
  • 1
  • 7
0
votes
0 answers

Spark GraphX - Implementing FoFoF

How can i find FoFoF in Spark GraphX ? In Cypher i have query: MATCH (f:Friend)-[:friend_of]-(Friend)-[:friend_of]-(c:Friend) WHERE c.name STARTS WITH "T" RETURN f.name AS Friend1 , c2.name as Friend2 Another example from here MATCH (john {name:…
Yehuda
  • 457
  • 2
  • 6
  • 16
0
votes
1 answer

Apache Spark Graphx - Java Implementation

As per Spark Documentation, it seems GraphX does not have Java API available yet. IS my assumption correct? If yes then can somebody provide some example where GraphX library is called using Java Code?
Sourav Gulati
  • 1,359
  • 9
  • 18
0
votes
1 answer

Strange bug with INT/LONG conversion when using Spark GraphX

New developer in Scala here and also a new user of Spark GraphX. So far, I'm really enjoying my time but I've just had a really strange bug. I have isolated the problem to a long-to-int conversion but it's really weird. Another weird thing is that…
toto
  • 880
  • 11
  • 21
0
votes
2 answers

Why does "spark-shell --jars" with GraphFrames jar give "error: missing or invalid dependency detected while loading class file 'Logging.class'"?

I have run a command spark-shell --jars /home/krishnamahi/graphframes-0.4.0-spark2.1-s_2.11.jar and it threw me an error error: missing or invalid dependency detected while loading class file 'Logging.class'. Could not access term typesafe in…
D Sai Krishna
  • 178
  • 1
  • 13
0
votes
1 answer

save page rank output in neo4j

I am running Pregel Page rank algorith m on twitter data in Spark using scala. The algorithm runs fine and gives me the output correctly finding out the highest page rank score. But I am unable to save graph on neo4j. The inputs and outputs are…
Chandni
  • 11
0
votes
1 answer

Is it OK to have nodes with mutable attributes when using Spark's GraphX distributed functions?

I am looking at the implementation of a certain graph clustering algorithm using Spark's GraphX graph analytics library. I noticed that the implementation uses a class VertexState with several mutable (var members). I wonder whether doing this sort…
Mateo
  • 1,494
  • 1
  • 18
  • 27
0
votes
1 answer

Spark GraphX - How to pass and array to to filter graph edges?

I am using Scala on Spark 2.1.0 GraphX. I have an array as shown below: scala> TEMP1Vertex.take(5) res46: Array[org.apache.spark.graphx.VertexId] = Array(-1895512637, -1745667420, -1448961741, -1352361520, -1286348803) If I had to filter the edge…
SoakingHummer
  • 562
  • 1
  • 7
  • 25
1 2 3
…
32 33