how to run topic model on 20000 documents at once?

Question

I have 20000 news documents to run topic modeling on it:

I want to see the topic dynamics and evolution from the documents. I tried to use the following batch script with Topic modeling by mallet but not work.

#!/bin/bash
for filename in /Users/JasonDou/code/internet_finance/bydocafterseg2; do
    ./bin/mallet import-dir --input /Users/JasonDou/code/internet_finance/bydocafterseg2/159047443.txt  --output bydoc-input.mallet --keep-sequence --remove-stopwords
done

score 1 · Accepted Answer · answered Jun 06 '16 at 08:36

1

You are missing an asterisk:

#!/bin/bash
for filename in "/Users/JasonDou/code/internet_finance/bydocafterseg2/"*; do
    [ -e "$filename" ] || continue
    ./bin/mallet import-dir --input "$filename" \
      --output bydoc-input.mallet --keep-sequence --remove-stopwords
done

The above will list iterate over each file in bydocafterseg2. You can change it to all .txt files with: "bydocafterseg2/"*".txt"

answered Jun 06 '16 at 08:36

Andreas Louv

46,145
13
104
123

it helps! I have a follow up question: how can I name the output file according the filename above? like "$filename"-input.mallet ? – Jason Jun 06 '16 at 13:14
`--output "$filename-input.mallet"`? – Andreas Louv Jun 06 '16 at 14:28

how to run topic model on 20000 documents at once?

1 Answers1