0

I am running Neo4j 2.0.2 on a Windows 2012 server and have a need to import hundreds of millions of nodes/relationships quickly (a day). Are there any bulk import tools or processes that can be used on windows? I have attempted to load multiple create statements using the spreadsheet method discussed on Neo4j's site but found that the statements just hang if I try to run more than 1000 statements at a time.

Gary Griggs
  • 1
  • 3
  • Are you doing a fresh load every day or are you adding hundreds of millions of nodes/relationships each day? – Kenny Bastani Apr 25 '14 at 20:48
  • This is a fresh load to create an example database so I can prove out some use cases. Going forward I can see a bulk loader being useful to do an initial population of a database but don't expect to perform these types of loads regularly. – Gary Griggs Apr 26 '14 at 02:00

1 Answers1

0

For high performant initial data population the best way is to use the batch inserter. This requires either some Java coding or use @mesiiri's tool to run batch inserter directly on csv files.

Also be sure to read his excellent blog post on http://jexp.de/blog/2013/05/on-importing-data-in-neo4j-blog-series/

Stefan Armbruster
  • 39,465
  • 6
  • 87
  • 97
  • @mesiiri's tool is exactly what I'm looking for. Where do you run the code? – Gary Griggs Apr 26 '14 at 12:17
  • `batch-import` requires java and maven to be run. It generates the graph-db directory contents for you. When done you can run your neo4j server based on that directory. – Stefan Armbruster Apr 26 '14 at 12:20
  • I have downloaded maven. Does the bulk importer files need to be in the maven directory? Their example execution line is mvn clean compile exec:java -Dexec.mainClass="org.neo4j.batchimport.Importer" -Dexec.args="neo4j/data/graph.db nodes.csv rels.csv" For this do you need to be at a command line in the maven directory? If so, the csv files will also need to be in the same directory? I appreciate your help with this. – Gary Griggs Apr 26 '14 at 21:20
  • No need to extract bulk importer inside maven directory. Put it anywhere locally, open a console for that directory and run the `mvn` command. For the csv files you can specify paths as well. – Stefan Armbruster Apr 27 '14 at 16:31