0

I have a 200GB RDF file in .nt format. I want to load it in Virtuoso (using Virtuoso Open-Source Edition 6.1.6). I used Virtuoso bulk loader from command line but loading gets hang after couple of hours of running. Do you have any idea how I can load this large file to Virtuoso efficiently? I want to load it fast.

I also tried to query my 200GB RDF graph from Apache Jena. However after running for 30 minutes it gives me some heap size space related error. If you have any solution for the above problem then kindly let me know.

Beautiful Mind
  • 5,828
  • 4
  • 23
  • 42
  • You'll need to say more about the setup. How many triples? For Jena, what storage are loading it into? (It will not fit in-memory.) How are you loading it into Jena? What hardware are you using? – AndyS Mar 25 '14 at 10:55

2 Answers2

0

Jena TDB has a bulk loader which has been used on large data input (hundred's of millions of triples).

AndyS
  • 16,345
  • 17
  • 21
0

What is the actual dataset you are loading? Is it actually just one file? We would recommend splitting into files of about 1GB max, and loading multiple files at a time with the bulk loader.

Have you done any performance tuning of the Virtuoso Server for the resources available on the machine in use, as detailed in the RDF Performance Tuning guide?

Please check with the status(''); command how many buffers are in use as, if you run out during a load, you will be swapping to disk continuously, which will lead to the sort of apparent hangs you report.

Note you can also load the Virtuoso LD Meter functions to monitor the progress of the dataset loads.

TallTed
  • 9,069
  • 2
  • 22
  • 37
HughWilliams
  • 126
  • 1