how to remove NA columns from TDM for clustering

Question

I'm struggling with TDM NA values to commit the clustering. Initially I've set:

titles.tdm <- as.matrix(TermDocumentMatrix(titles.cw, control = list(bounds = list(global = c(10,Inf)))))

titles.sc <- scale(na.omit(titles.tdm))

and got matrix of 418 terms and 6955 documents. At this point executing: titles.km <- kmeans(titles.sc, 2) throws Error in do_one(nmeth) : NA/NaN/Inf in foreign function call (arg 1)

When I've decided to remove those values by:

titles.sf <- titles.sc[,colSums(titles.sc) > 0]

I've got matrix of 4695 documents, but applying the kmeans function still throws this error. When I've viewed the titles.sf variable there are still columns (docs) with NA values. I'm messed up and don't know what doing wrong. How to remove those documents?

Earlier, I've applied titles.cw <- titles.cc[which(str_trim(titles.cc$content) != "")] where titles.cc is pure Corpus object from tm library class, to delete black documents. It probably worked, but my NA values are in documents which are not blank for sure.

There shouldn't be any NA values in a correct TDM. – Has QUIT--Anony-Mousse May 01 '17 at 18:12 — Has QUIT--Anony-Mousse, May 01 '17 at 18:12

score 2 · Accepted Answer · answered Apr 30 '17 at 23:02

Here's some example data:

set.seed(123)
titles.sc <- matrix(1:25,5,5)
titles.sc[sample(length(titles.sc),5)]<-NA 
titles.sc
#      [,1] [,2] [,3] [,4] [,5]
# [1,]    1    6   11   16   21
# [2,]    2    7   12   17   NA
# [3,]    3   NA   13   18   23
# [4,]    4    9   14   NA   24
# [5,]    5   NA   15   NA   25

kmeans throws your error

kmeans(titles.sc, 2)
# Error in do_one(nmeth) : NA/NaN/Inf in foreign function call (arg 1)

because your column subsetting is probably not what you'd expect:

colSums(titles.sc) > 0
# [1] TRUE   NA TRUE   NA   NA

colSums produces NA, if missing values are not removed (check the help files under ?colSums). Among other things, you could do

colSums(is.na(titles.sc)) == 0
# [1]  TRUE FALSE  TRUE FALSE FALSE

or

!is.na(colSums(titles.sc) > 0)
# [1]  TRUE FALSE  TRUE FALSE FALSE

And now, it works:

titles.sf <- titles.sc[,colSums(is.na(titles.sc)) == 0,drop=F]
kmeans(titles.sf,2)
# K-means clustering with 2 clusters of sizes 2, 3
# ...

how to remove NA columns from TDM for clustering

1 Answers1