How to insert large number of nodes into Neo4J

Question

I need to insert about 1 million of nodes in Neo4j. I need to specify that each node is unique, so every time I insert a node it has to be checked that there's not the same node yet. Also the relationships must be unique.

I'm using Python and Cypher:

uq = 'CREATE CONSTRAINT ON (a:ipNode8) ASSERT a.ip IS UNIQUE'
...
queryProbe = 'MERGE (a:ipNode8 {ip:"' + prev + '"})'
...
queryUpdateRelationship= 'MATCH (a:ipNode8 {ip:"' + prev + '"}),(b:ipNode8 {ip:"' + next + '"}) MERGE (a)-[:precede]->(b)'

The problem is that after putting 40-50K nodes into Neo4j , the insertion speed slows down quickly and I can not to put anything else.

Answer 1

Your question is quite open ended. In addition to @InverseFalcon's recommendations, here are some other things you can investigate to speed things up.

1. Read the Performance Tuning documentation, and follow the recommendations. In particular, you might be running into memory-related issues, so the Memory Tuning section may be very helpful.

2. Your Cypher query(ies) can probably be sped up. For instance, if it makes sense, you can try something like the following. The data parameter is expected to be a list of objects having the format {a: 123, b: 234}. You can make the list as long as appropriate (e.g., 20K) to avoid running out of memory on the server while it processes the list within a single transaction. (This query assumes that you also want to create b if it does not exist.)

   UNWIND {data} AS d
   MERGE (a:ipNode8 {ip: d.a})
   MERGE (b:ipNode8 {ip: d.b})
   MERGE (a)-[:precede]->(b)
   

   There are also periodic execution APOC procedures that you might be able to use.

Answer 2

For mass inserts like this, it's best to use LOAD CSV with periodic commit or the import tool.

I believe it's also best practice to use a parameterized query instead of appending values into a string.

Also, you created a unique property constraint on :ipNode8, but not :ipNode, which is the first one you MERGE. Seems like you'll need a unique constraint for that one too.