Stack Overflow
Stack Overflow

Questions tagged [spark-graphx]

GraphX is a component in Apache Spark for graphs and graph-parallel computation

GraphX is a component in Apache Spark for graphs and graph-parallel computation.

At a high level, GraphX extends the Spark RDD by introducing a new Graph abstraction: a directed multigraph with properties attached to each vertex and edge.

To support graph computation, GraphX exposes a set of fundamental operators (e.g., subgraph, joinVertices, and aggregateMessages) as well as an optimized variant of the Pregel API.

In addition, GraphX includes a growing collection of graph algorithms and builders to simplify graph analytics tasks.

487 questions
0
votes
2 answers

constructor cannot be instantiated to expected type; found : (T1, T2, T3, T4, T5)

I am new in spark-graphx and scala. I write this method to verify if the score of the goal vertex has changed. def IsGoalVertexFound(graph: Graph[(VertexId,(Int,Float,Float,Float,String)),Float],goalVertex:(VertexId,(Int,Float,Float,Float))):…
imen
  • 35
  • 8
0
votes
0 answers

overloaded method value + with alternatives:

I am new in scala and spark-graphx. I write this method to update the scores of the different neighbors of a vertex. def updateNeighborScores(triplet: EdgeTriplet[(VertexId,(Int,Float,Float,Float,String)),Float]):Float={ …
imen
  • 35
  • 8
0
votes
1 answer

GraphX-Spark: error graph.vertices.filter

I am new in scala and spark-graphx. This is a method that I've written to extract the vertex that has min score value def getMinScoreVertex(graph: Graph[(Int,Float,Float,Float,String),Float]):Float={ var minValue:Float=Float.PositiveInfinity var…
imen
  • 35
  • 8
0
votes
1 answer

Spark GraphX Out of memory error

I am running GraphX on Spark with input file size of around 100GB on aws EMR. My cluster configuration is as follows Nodes - 10 Memory - 122GB each HDD - 320GB each No matter what I do I'm getting out of memory error when I run spark job…
hlagvankar
  • 219
  • 1
  • 3
  • 12
0
votes
0 answers

How to convert to VertexId in Spark/Scala Graphx?

I am trying to create a graph in Apache Spark Graphx using Scala. My code so far is: import org.apache.spark._ import org.apache.spark.graphx._ case class EdgesCl(Source:Long, Destination:Long, SourceLayer:Long, DestLayer:Long,…
laz.koiou
  • 1
  • 3
0
votes
1 answer

how to declare graphX variable as global in scala

I am new in scala and spark-graphX. How can I declare spark-graphx variable as global variable in scala. I have a method to create graph with spark-graphX and it is successful. Now , I want to retrieve graph details from another method. For that, I…
Rhea
  • 381
  • 1
  • 7
  • 22
0
votes
0 answers

Call SparkSession in worker (Spark-SQL, Java)

I'm working with GraphX and SparkSQL and I'm trying to create DataFrame (Dataset) in a graph node. To create a DataFrame I need the SparkSession (spark.createDataFrame(rows,schema)).All I try, I get an error. This is my Code: SparkSession spark =…
Vitali D.
  • 149
  • 2
  • 14
0
votes
1 answer

Apache Spark GraphX using Cloudera Quickstart 5.7

I'm trying to use Spark GraphX using Cloudera Quickstart 5.7 from this website: http://kukuruku.co/hub/algorithms/social-network-analysis-spark-graphx The steps that I did: 1) Open Terminal 2) Run spark-shell 3) Run import…
João_testeSW
  • 99
  • 1
  • 12
0
votes
0 answers

Spark DataFrame to RDD - graph.apply with GraphX

I have a vertices and edges file in order to construct a graph using the apply method with GraphX. My input files are in json.gz format and I am thus using the spark sqlContext.read.json function to import the data. val vertices =…
LearningSlowly
  • 8,641
  • 19
  • 55
  • 78
0
votes
0 answers

Loading CSV into GraphX with Vertex Attributes

I'm new to GraphX and I'm trying to load a CSV file as a graph. The CSV is in the format: ID1 ID2 Attr1 Attr2 OtherAttr1 OtherAttr2 0 6 -2 4 soccer basketball 3 1 5 …
Aditya Srinivasan
  • 41
  • 2
0
votes
1 answer

different types of edges in graphx

How can I add different types of edges in the same graphX graph. For instance and edge between node A and node B may have String attribute and an edge between node X and node Y may have (Int, Int) attr. Is there a way to implement edge inheritance…
Mayuri M.
  • 121
  • 1
  • 2
0
votes
1 answer

How to convert, in Scala, an Array[VertexIds] to a Map?

In Scala, I have a an array of VertexIds v: Array[org.apache.spark.graphx.VertexId] = Array(-2634311911308936962, 2326575714372975825, ...) that i want to convert to a map where Map(-2634311911308936962 -> 0, 2326575714372975825 -> 1, …
Eoin Lane
  • 641
  • 2
  • 6
  • 22
0
votes
2 answers

Creating a graph in Spark from a time series table

Suppose I have a table with three columns; user, time, place. I want to create a graph of the place transitions for each user if the time between them is below a certain threshold; i.e., after grouping by user and ordering by time, create a directed…
Emre
  • 5,976
  • 7
  • 29
  • 42
0
votes
1 answer

Graphx get vertex label from vertex id

I have the following graph in Graphx graph.vertices.foreach(println) (6109253945443866644,"Futurama"@en) (7558506336564503178,"AccessibleComputing"@en) (0,null) (-2278222762001827643,"Programming…
user3663737
  • 3
  • 7
0
votes
1 answer

How would we get the whole edge which has the max weight in a graphX graph?

I'd like to know how to get the whole edge which has the max weight(attr) in a graphx graph? eg: | srcID | dstID | attr | ------------------------ | 3 | 7 | 2 | ------------------------ | 4 | 7 | 8 | ------------------------ |…
Mayuri M.
  • 121
  • 1
  • 2
1 2 3
…
32 33