1

I am trying to load a dataset in LibSVM Format with weka. Here the code:

LibSVMLoader svmLoader = new LibSVMLoader();
svmLoader.setSource(new File("data_with_libsvm_format"));
Instances svmData = svmLoader.getDataSet();
System.out.println(svmData.toSummaryString());

And I get the exception: 

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

I set -Xmx4000m and notice that the program rapidly consumes 4000m of RAM and throws exception. Does Weka really handle libsvm format or load and store the data in dense format internally during the process?

MWiesner
  • 8,868
  • 11
  • 36
  • 70
Duong Trung Nghia
  • 310
  • 4
  • 13
  • describe the input data - LibSVM can be mem consuming, much depends on the data you feed in. – MWiesner Aug 15 '15 at 16:00
  • if that, the LibSVMLoader in Weka is not efficient to handle large LibSVM dataset. Do you have any ideas why is that? I mean how Weka really handles libsvm format or load and store the data in dense format internally during the process? – Duong Trung Nghia Aug 17 '15 at 18:44
  • I don't know what went wrong with Weka. But I tried scikit-learn and coded in Python on the same dataset. It went well for me. – Duong Trung Nghia Oct 15 '15 at 18:59

0 Answers0