2

I use quanteda to build a document term matrix:

library(quanteda)
mytext = "This is my old text"
dtm <- dfm(mytext, tolower=T)
convert(dtm,to="data.frame")

Which yields:

  doc_id this is my old text
1  text1    1  1  1   1    1

I need to fit "new" text (a new corpus) to my existing dtm (using the same vocabulary so that the same matrix columns will be present)

Suppose my "new" text/corpus would be:

newtext = "This is my new text"

How can I fit this "new" text/corpus to the existing dtm vocabulary, so to get a matrix like:

  doc_id this is my old text
1  text1    1  1  1   0    1
Peter
  • 2,120
  • 2
  • 19
  • 33

1 Answers1

2

You want dfm_match(), before converting to data.frame.

library(quanteda)
## Package version: 2.1.2

mytext <- c(oldtext = "This is my old text")
dtm_old <- dfm(mytext)
dtm_old
## Document-feature matrix of: 1 document, 5 features (0.0% sparse).
##          features
## docs      this is my old text
##   oldtext    1  1  1   1    1

newtext <- c(newtext = "This is my new text")
dtm_new <- dfm(newtext)
dtm_new
## Document-feature matrix of: 1 document, 5 features (0.0% sparse).
##          features
## docs      this is my new text
##   newtext    1  1  1   1    1

To match them up, use dfm_match() to conform the new dfm to the feature set and order of the old one:

dtm_matched <- dfm_match(dtm_new, featnames(dtm_old))
dtm_matched
## Document-feature matrix of: 1 document, 5 features (20.0% sparse).
##          features
## docs      this is my old text
##   newtext    1  1  1   0    1

convert(dtm_matched, to = "data.frame")
##    doc_id this is my old text
## 1 newtext    1  1  1   0    1
Ken Benoit
  • 14,454
  • 27
  • 50
  • Thanks! We designed this for matching the feature set of out-of-sample data to training data, which is I'm guessing your use case too. – Ken Benoit Nov 11 '20 at 19:02
  • 1
    Jup, exactly. I'm quite happy with quanteda! I like the "on the fly" text preprocessing. I'm new to the package, but so far impressed... Very good work! – Peter Nov 11 '20 at 19:32