2

I have build a property graph(60 million nodes, 40 million edges) from s3 using Apache Spark Graphx framework. I want to fire traversal queries on this graph.

My queries will be like:-

g.V().has("name","xyz").out('parent').out().has('name','abc')

g.V().has('proc_name','serv.exe').out('file_create').
has('file_path',containing('Tsk04.txt')).in().in('parent').values('proc_name')

g.V().has('md5','935ca12348040410e0b2a8215180474e').values('files')

mostly queries are of form g.V().out().out().out()

Such queries are easily possible on graph db's like neo4j,titan,aws neptune since they support gremlin.

Can we traverse spark graphs in such manner. I tried spark pregel-api but its bit complex as compared to gremlin.

Reason I am looking for spark graph is because cloud solutions of above mentioned graphdbs is costly.

Stanislav Kralin
  • 11,070
  • 4
  • 35
  • 58
AbhiK
  • 247
  • 3
  • 19

1 Answers1

3

Spark GraphFrames library should be most convenient for you. it provides neo4j-cypher-like traversal description and use Spark DataFrames api for filtering
https://graphframes.github.io/graphframes/docs/_site/user-guide.html#motif-finding Here is an example:

val g2: GraphFrame = GraphFrame.fromGraphX(gx) // you can start with just V and E DataFrames here
val motifs: GraphFrame = g.find("(a)-[e]->(b); (b)-[e2]->(c)")
motifs.filter("a.name = 'xyz'  and e.label = 'parent' and c.name = 'abc'").show()

TinkerPop it self has spark support, so you can issue spark OLAP queries from gremlin console https://tinkerpop.apache.org/docs/current/reference/#sparkgraphcomputer

Or there are some close source solutions. Datastax Enterprise Database has a good Gremlin support for spark: https://www.datastax.com/blog/2017/05/introducing-dse-graph-frames I'm a former author of it

Douglas M
  • 1,035
  • 8
  • 17
Artem Aliev
  • 1,362
  • 7
  • 12
  • Thanks a lot @Artem Aliev explored GraphFrame it feels like I can solve most of my use-cases using it. – AbhiK May 29 '20 at 18:48