Mallet: OutOfMemoryError: Java heap space

Question

While training data in Mallet, the processed stopped because of OutOfMemoryError. Attribute MEMORY in bin/mallet has already been set to 3GB. The size of training file output.mallet is only 31 MB. I have tried to reduce the training data size. But it still throws the same error:

a161115@a161115-Inspiron-3250:~/dev/test_models/Mallet$ bin/mallet train-classifier --input output.mallet --trainer NaiveBayes --training-portion 0.0001 --num-trials 10
Training portion = 1.0E-4
Unlabeled training sub-portion = 0.0
Validation portion = 0.0
Testing portion = 0.9999

-------------------- Trial 0  --------------------

Trial 0 Training NaiveBayesTrainer with 7 instances
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at cc.mallet.types.Multinomial$Estimator.setAlphabet(Multinomial.java:309)
        at cc.mallet.classify.NaiveBayesTrainer.setup(NaiveBayesTrainer.java:251)
        at cc.mallet.classify.NaiveBayesTrainer.trainIncremental(NaiveBayesTrainer.java:200)
        at cc.mallet.classify.NaiveBayesTrainer.train(NaiveBayesTrainer.java:193)
        at cc.mallet.classify.NaiveBayesTrainer.train(NaiveBayesTrainer.java:59)
        at cc.mallet.classify.tui.Vectors2Classify.main(Vectors2Classify.java:415)

I would appriciate any help or insights into this problem

EDIT: this is my bin/mallet file.

#!/bin/bash


malletdir=`dirname $0`
malletdir=`dirname $malletdir`

cp=$malletdir/class:$malletdir/lib/mallet-deps.jar:$CLASSPATH
#echo $cp

MEMORY=10g

CMD=$1
shift

help()
{
cat <<EOF
Mallet 2.0 commands: 

  import-dir         load the contents of a directory into mallet instances (one per file)
  import-file        load a single file into mallet instances (one per line)
  import-svmlight    load SVMLight format data files into Mallet instances
  info               get information about Mallet instances
  train-classifier   train a classifier from Mallet data files
  classify-dir       classify data from a single file with a saved classifier
  classify-file      classify the contents of a directory with a saved classifier
  classify-svmlight  classify data from a single file in SVMLight format
  train-topics       train a topic model from Mallet data files
  infer-topics       use a trained topic model to infer topics for new documents
  evaluate-topics    estimate the probability of new documents under a trained model
  prune              remove features based on frequency or information gain
  split              divide data into testing, training, and validation portions
  bulk-load          for big input files, efficiently prune vocabulary and import docs

Include --help with any option for more information
EOF
}

CLASS=

case $CMD in
        import-dir) CLASS=cc.mallet.classify.tui.Text2Vectors;;
        import-file) CLASS=cc.mallet.classify.tui.Csv2Vectors;;
        import-svmlight) CLASS=cc.mallet.classify.tui.SvmLight2Vectors;;
        info) CLASS=cc.mallet.classify.tui.Vectors2Info;;
        train-classifier) CLASS=cc.mallet.classify.tui.Vectors2Classify;;
        classify-dir) CLASS=cc.mallet.classify.tui.Text2Classify;;
        classify-file) CLASS=cc.mallet.classify.tui.Csv2Classify;;
        classify-svmlight) CLASS=cc.mallet.classify.tui.SvmLight2Classify;;
        train-topics) CLASS=cc.mallet.topics.tui.TopicTrainer;;
        infer-topics) CLASS=cc.mallet.topics.tui.InferTopics;;
        evaluate-topics) CLASS=cc.mallet.topics.tui.EvaluateTopics;;
        prune) CLASS=cc.mallet.classify.tui.Vectors2Vectors;;
        split) CLASS=cc.mallet.classify.tui.Vectors2Vectors;;
        bulk-load) CLASS=cc.mallet.util.BulkLoader;;
        run) CLASS=$1; shift;;
        *) echo "Unrecognized command: $CMD"; help; exit 1;;
esac

java -Xmx$MEMORY -ea -Djava.awt.headless=true -Dfile.encoding=UTF-8 -server -classpath "$cp" $CLASS "$@"

It's also worth mentioning that my original training file has 60,000 items. When I reduce the number of items (20,000 instances), training will run like normal, but uses about 10GB RAM.

Which file exactly did you change it in? bin/mallet or bin/mallet.sh? — mikep, Jun 22 '17 at 05:18
Check my answer. That will increase the java heap size. If MEMORY appears in the java call, replace it with 3g. — mikep, Jun 22 '17 at 08:16
Thank you for your help. I have posted my file above. I don't see any duplicate Xmx though. I think the problem might be with the training file I have. It has too many instances. I was wondering whether there is a way to train one instance at a time and then release memory from mallet? — Long Le Minh, Jun 23 '17 at 05:33

mikep · Answer 1 · 2017-06-22T08:15:23.257

1

Check the call to Java in bin/mallet and add the flag -Xmx3g, making sure there isn't another Xmx in it; if so, edit that one).

edited Jun 22 '17 at 08:15

answered Jun 22 '17 at 04:43

mikep

3,841
8
21

score 0 · Answer 2 · answered Sep 20 '18 at 12:11

I usually change both files: mallet files and set the memory to maximum

Mallet.batjava -Xmx%MALLET_MEMORY% -ea -Dfile.encoding=%MALLET_ENCODING% -classpath %MALLET_CLASSPATH% %CLASS% %MALLET_ARGS%

and

java -Xmx$MEMORY -ea -Djava.awt.headless=true -Dfile.encoding=UTF-8 -server -classpath "$cp" $CLASS "$@"

I replaced the bold %MALLET_MEMORY% and $MEMORY with the memory I want: e.g. 4G

Mallet: OutOfMemoryError: Java heap space

2 Answers2