Neo4j (Windows) - can't increase heap memory size for Neo4jImport tool

Question

I tried to batch import a graph database with about 40 million nodes and 20 million relationships but I get an outofmemory error (this has been documented already, I know). On Windows, I am using the import tool as so:

neo4jImport –into SemMedDB.graphdb --nodes nodes1.csv --nodes nodes2.csv --relationships edges.csv

I have 16 GB of RAM but Neo4j only allocates 3.5 GB of max heap memory while I still have about 11 GB of free RAM. To try to fix this so I wouldn't get an outofmemory error, I followed some suggestions online and created a conf folder in my C:\program files\Neo4j folder and created a neo4j-wrapper.conf file with heap values set to:

wrapper.java.initmemory=10000
wrapper.java.maxmemory=10000

Also, I set my neo4j properties file page cache setting to:

dbms.pagecache.memory=5g

The problem is, when I restart my neo4j application and try to import again, it still says 3.5 GB of max heap space and 11 GB free RAM... why doesn't Neo4j recognize my settings?

Note, I've tried downloading the zip version of Neo4j in order to use the powershell version of the import tool but I run into the same issue of changing my configuration settings but Neo4j not recognizing them.

I would really appreciate some help with this... thanks!

score 2 · Answer 1 · answered Feb 06 '16 at 09:33

2

Cannot tell for windows, but on linux neo4j-wrapper.conf is not used for neo4j-import tool. Instead you can pass additional JVM parameters using JAVA_OPTS environment variable (again Linux syntax here):

JAVA_OPTS="-Xmx10G" bin/neo4j-import

To validate that approach, amend -XX:+PrintCommandLineFlags to the above. At the beginning of the output you should see a line similar to

-XX:InitialHeapSize=255912576 -XX:MaxHeapSize=4094601216 \n
-XX:+PrintCommandLineFlags -XX:+UseCompressedClassPointers \n
-XX:+UseCompressedOops -XX:+UseParallelGC

If that one shows up, using JAVA_OPTS is the way to go.

answered Feb 06 '16 at 09:33

Stefan Armbruster

39,465
6
87
97

What shell am I supposed to use JAVA_OPTS in? I opened neo4j.exe, went to the command prompt, and used your command (w/ my path of course) but it gave the error "JAVA_OPTS is not recognized as an internal or external command"... – Sergei Wallace Feb 06 '16 at 15:58
As mentioned the sketched approach is taken from Linux. Not sure how that works on Windows. – Stefan Armbruster Feb 06 '16 at 16:27
Right, but you would essentially do the same thing in Linux? Open the Neo4j shell and type the JAVA_OPTS command? – Sergei Wallace Feb 06 '16 at 16:36
1

Nope. JAVA_OPTS needs to be set e.g. in cmd.exe before invoking neo4j-import – Stefan Armbruster Feb 06 '16 at 16:39
I see, the command executed now. I used the commandLineFlags argument too but I didn't get any output? – Sergei Wallace Feb 06 '16 at 17:05

score 1 · Accepted Answer · answered Feb 06 '16 at 19:00

I found a solution. What ultimately allowed me to change the heap size for the Neo4jImport tool was to open the neo4jImport.bat file (path is C:Program files\neo4j\bin) in a text editor (required me to changed permissions first) and change the "set EXTRA_JVM_ARGUMENTS=-Dfile.encoding=UTF-8" line to

set EXTRA_JVM_ARGUMENTS=-Dfile.encoding=UTF-8 -Xmx10G -Xms10G -Xmn2G

Now, when I run Neo4jImport in the neo4j shell, it shows a heap size of 9.75 GB.

score 1 · Answer 3 · answered Feb 09 '16 at 12:32

Generally Neo4jImport shouldn't realy on a large heap, it will use whatever heap is available and then use whatever offheap is available, however some amount of "boilerplate" memory needs to be there for the machinery to work properly. Recently there were a fix (coming in 2.3.3) reducing heap usage of the import tool and that would have certainly helped here.

Neo4j (Windows) - can't increase heap memory size for Neo4jImport tool

3 Answers3