3

If I do something like this:

from py2neo import Graph
graph = Graph()
stuff = graph.cypher.execute("""
    match (a:Article)-[p]-n return a, n, p.weight
""")

on a database with lots of articles and links, the query takes a long time and uses all my system's memory, presumably because it's copying the entire result set into memory in one go. Is there some kind of cursor-based version where I could iterate through the results one at a time without having to have them all in memory at once?

EDIT

I found the stream function:

stuff = graph.cypher.stream("""
    match (a:Article)-[p]-n return a, n, p.weight
""")

which seems to be what I want according to the documentation but now I get a timeout error (py2neo.packages.httpstream.http.SocketError: timed out), followed by the server becoming unresponsive until I kill it using kill -9.

Andrew Magee
  • 6,506
  • 4
  • 35
  • 58

1 Answers1

2

Have you tried implementing a paging mechanism? Perhaps with the skip keyword: http://neo4j.com/docs/stable/query-skip.html

Similar to using limit / offset in a postgres / mysql query.

EDIT: I previously said that the entire result set was stored in memory, but it appears this is not the case when using api streaming - per Nigel's (Neo engineer) comment below.

Paul Shoemaker
  • 136
  • 3
  • Yeah I did think of that but I'm hoping not to have to; seems like an unwieldy way of implementing something so simple. – Andrew Magee Feb 17 '15 at 22:57
  • 1
    The stream method does not place the entire result in memory - it incrementally parses the JSON returned. – Nigel Small Feb 18 '15 at 00:54
  • @NigelSmall - are you saying that the result set is not held in memory on the graph database machine? My apologies on misunderstanding how this works ... – Paul Shoemaker Feb 19 '15 at 01:57
  • When using the `stream` method, the results are both streamed from the server as they are calculated (http://neo4j.com/docs/stable/rest-api-streaming.html) and streamed from the response on the client as it is received into the calling application (https://github.com/nigelsmall/jsonstream). – Nigel Small Feb 19 '15 at 10:56
  • @NigelSmall - wicked! Thanks for clarifying. I'll edit my comment to reflect the truth. – Paul Shoemaker Feb 19 '15 at 20:11