1

I am trying to create an R-TREE of 5834580 of nodes.
I found in this question a similair problem and i tried it's solution, so this is my code :

call apoc.periodic.commit("MATCH (pl:pickup_location) WITH collect(pl) AS pickup CALL spatial.addNodes('nyc',pickup) YIELD count RETURN count",{limit:1000})

however, since yesterday the computer didn't finish loading the result.

today, i tried the second answer with iterate :

CALL apoc.periodic.iterate(
"MATCH (pl:pickup_location) RETURN pl",
"CALL spatial.addNode('nyc', pl) YIELD node RETURN node",
{batchSize:10000, parallel:false, listIterate:true})

and i get this error :

Neo.ClientError.Procedure.ProcedureCallFailed: Failed to invoke procedure `apoc.periodic.iterate`: Caused by: java.lang.OutOfMemoryError: Java heap space

what is wrong ? what should i do ?

A.HADDAD
  • 1,809
  • 4
  • 26
  • 51

2 Answers2

2

Your problem with the apoc.periodic.commit is that your query will always return the same nodes with your MATCH (pl:pickup_location). There is no condition to find only nodes that are not in the spatial layout.

I don't remember the model of the spatial plugin, but from what I remember, on your pickup_location nodes, you should have a specific relation to the R-Tree.

So you should transform your auery to something like that : 

CALL apoc.periodic.commit("
  MATCH (pl:pickup_location)
  WHERE NOT (p1)-[:LINKS->(:spatialNode) // Change this according to the spatial model
  WITH p1 AS node LIMIT $limit
  WITH collect(node) AS pickup 
    CALL spatial.addNodes('nyc',pickup) YIELD count 
    RETURN count",
  {limit:1000}
)

For the problem on the apoc.periodic.iterate is just a memory problem, you don't have enought RAM to execute transactions.

You have two solutions :

  • give more RAM to Neo4j by increasing the heap size of Neo4j (see the neo4j.conf file)
  • decrease the size of the batch, 10000 is a little big, change it to 1000
logisima
  • 7,340
  • 1
  • 18
  • 31
  • I don't see usage of `limit` within the query for `commit()`. Unlike `apoc.periodic.iterate()`, you must explicitly add the usage of `limit`. With this query in particular the location of the `limit` is important, since you're collecting down to 1 row, so for this to be effective you need to have `WITH pl LIMIT {limit}` before the WITH clause that performs the collect(). – InverseFalcon Jun 25 '18 at 03:01
  • I have updated my response :) Thanks the point me this missing part – logisima Jun 25 '18 at 08:44
1

As I don't have a rep of 50 I can't comment but logisima is spot on, just to add to their answer... the "where not" clause should be:

where not (p1)-[:RTREE_REFERENCE]-()

RTREE_REFERENCE being the relationship created when adding a node to the spatial layer.

SAB
  • 175
  • 17