4

I'm trying to import 130M nodes and 200M relationships on a new neo4j database from csv files using neo4-import tool and it gets stuck on the process.

The output of the process:

Nodes [>:3.35 MB/s---|PROPERTIES(2)======|*NODE:1007.08 MB------|LABEL SCAN-----|v:7.46 MB/s(2)======]131M
Done in 1h 35m 47s 332ms
Prepare node index
[*SORT:1.47 GB---------------------------------------------------------------------------------] 87M

The content of messages.log

2015-06-15 10:38:56.423+0000 INFO  [org.neo4j]: Import starting
2015-06-15 10:38:56.537+0000 INFO  [org.neo4j]: Creating new db @ /var/user/neo/graph22.db/neostore
2015-06-15 10:38:57.264+0000 INFO  [org.neo4j]: Missing counts store, rebuilding it.
2015-06-15 10:38:57.536+0000 INFO  [org.neo4j]: No lucene scan store index found, this might just be first use. Preparing to rebuild.
2015-06-15 10:38:57.581+0000 INFO  [org.neo4j]: No lucene scan store index found, this might just be first use. Preparing to rebuild.
2015-06-15 10:38:57.793+0000 INFO  [org.neo4j]: Rebuilding lucene scan store, this may take a while
2015-06-15 10:38:57.796+0000 INFO  [org.neo4j]: Lucene scan store rebuilt (roughly -1 nodes)
2015-06-15 18:29:33.509+0000 INFO  [org.neo4j]: About to rotate counts store at transaction 1 to [/var/user/neo/graph22.db/neostore.counts.db.b], from [/var/user/neo/graph22.db/neostore.counts.db.a].
2015-06-15 18:29:33.516+0000 INFO  [org.neo4j]: Successfully rotated counts store at transaction 1 to [/var/user/neo/graph22.db/neostore.counts.db.b], from [/var/user/neo/graph22.db/neostore.counts.db.a].
2015-06-15 18:29:33.771+0000 ERROR [org.neo4j]: Error during import null
java.lang.StackOverflowError
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.partition(ParallelSort.java:193)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:243)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)
        at org.neo4j.unsafe.impl.batchimport.cache.idmapping.string.ParallelSort.recursiveQsort(ParallelSort.java:246)

What's causing the process to fail? Is there anything i can do in order to solve this?

I'm using neo4j-community-2.2.2

Adrian Fdz.
  • 41
  • 4
  • Which 2.2.x version are you using? – Mattias Finné Jun 16 '15 at 13:31
  • I'm using neo4j-community-2.2.2 – Adrian Fdz. Jun 16 '15 at 15:23
  • How does your node IDs look generally and which --id-type are you using? – Mattias Finné Jun 16 '15 at 19:40
  • Default value for --id-type (STRING). The node IDs are base64 encoded Strings without equals '=' and new line characters '\n'. Some of them may contain underline chars '_' – Adrian Fdz. Jun 16 '15 at 20:07
  • Have you tried running the import again? Is this error reproducible? – Mattias Finné Jun 17 '15 at 14:35
  • Yes, It always gets stuck on `[*SORT:1.47 GB---------------------------------------------------------------------------------] 87M` – Adrian Fdz. Jun 18 '15 at 06:01
  • I'm having the same issue, trying to import a 75M file – PedroSena Jun 18 '15 at 18:48
  • Strange that it's so consistent since there's an element of randomness in the sorting algorithm. See https://github.com/neo4j/neo4j/pull/4877 for a probable improvement though. Also getting access to the raw data would be terribly helpful in debugging this issue. Thanks in advance. – Mattias Finné Jun 22 '15 at 20:11
  • I can't share the data with you because it is sensitive. I am testing the importer with a subset of the data and so far it is working properly. could it be related to the size of the data? – Adrian Fdz. Jun 23 '15 at 06:48
  • It probably is. You mentioned it gets stuck, is it actually getting stuck or fails with the StackOverflowError. Do both happened randomly there? How long have you waited? – Mattias Finné Jun 25 '15 at 11:29
  • It gets stuck for hours (87M on 'Prepare node index') and then it fails with the StackOverflowError. Is there anything I can check in order to ensure that is a size related problem? – Adrian Fdz. Jun 26 '15 at 12:49
  • There's a PR to change the implementation to an iterative qsort instead to avoid stackoverflow, https://github.com/neo4j/neo4j/pull/4877 . That should improve things for you. – Mattias Finné Jul 12 '15 at 21:07

1 Answers1

2

I had the same problem, but with the version 2.2.3. I kind of solved it by editing the file bin/neo4j-import and adding more stack space (and more heap as well, just because i have it):

EXTRA_JVM_ARGUMENTS="-Dfile.encoding=UTF-8 -Xmx8g -Xss128m"

But now I have a different problem - recursion is 1023 levels deep (and I am only sorting around 500 million nodes), and I am not sure if the sort will ever finish :)

Alexey Akhunov
  • 121
  • 2