2

I asked this question before and got negative feedback because I did not provide code. I spent the whole day trying and trying and now I am stuck in an issue.

This code has been fetched by a user in Stackoverflow "Tyler Rincker" <- big thanks to him!

here is the code:

strip <- function(x, digit.remove = TRUE, apostrophe.remove = FALSE){
          strp <- function(x, digit.remove, apostrophe.remove){
            x2 <- Trim(tolower(gsub(".*?($|'|[^[:punct:]]).*?", "\\1", as.character(x))))
            x2 <- if(apostrophe.remove) gsub("'", "", x2) else x2
            ifelse(digit.remove==TRUE, gsub("[[:digit:]]", "", x2), x2)
          }
          unlist(lapply(x, function(x) Trim(strp(x =x, digit.remove = digit.remove, 
                              apostrophe.remove = apostrophe.remove)) ))

corpus2 <- "In Westerman's disruptive article, Quantitative research as 
        an interpretive enterprise: The mostly 
        unacknowledged role of interpretation in research efforts."

    corpus2 <- gsub("\\s+", " ", gsub("\n|\t", " ", corpus2))
    corpus2.wrds <- as.vector(unlist(strsplit(strip(corpus2), " ")))

    corpus2.Freq <- data.frame(table(corpus2.wrds))
    corpus2.Freq$corpus2.wrds  <- as.character(corpus2.Freq$corpus2.wrds)
    corpus2.Freq <- corpus2.Freq[order(-corpus2.Freq$Freq), ]
    rownames(corpus2.Freq) <- 1:nrow(corpus2.Freq)

    key.terms <- c("research as")

My issue is that i want to search for bigrams or trigram (2 or 3 words) in the corpus.

When i execute this line of code:

corpus2.Freq[corpus2.Freq$corpus2.wrds %in%key.terms, ]

I get this results which should show a frequency of "1".

[1] corpus2.wrds Freq        
<0 rows> (or 0-length row.names)

However, if the keyterm is only 1 word:

key.terms <- c("research")
    corpus2.Freq[corpus2.Freq$corpus2.wrds %in%key.terms, ]

the code is working fine and i get the following result:

corpus2.wrds Freq
research    2

Thanks a lot! and hopefully someone can help.

Lian Ahmad
  • 109
  • 1
  • 10

1 Answers1

1

you cannot find any bigrams, because you haven't created them. You use function strsplit(strip(corpus2), " ") which creates only unigrams.

I also recommend you to use preprocessing functions which are available from tm package instead of your own strip() function.

You can try something like this for unigrams :

library(tm)

# create corpora
corp = VCorpus(VectorSource(corpus2)) 

# this should replace your strip() function
ctrl = list(tokenize = function(x) unlist(strsplit(as.character(x), "[[:space:]]+")),
        removePunctuation = TRUE,
        removeNubers = TRUE,
        tolower = TRUE)

tdm = TermDocumentMatrix(corp, control = ctrl)
inspect(tdm)
inspect(tdm[c("research"),])

Or similar code for both unigrams and bigrams:

ctrl2 = list(tokenize = function(x) unlist(lapply(ngrams(words(x), 1:2), paste, collapse = " "), use.names = FALSE),
        removePunctuation = TRUE,
        removeNubers = TRUE,
        tolower = TRUE)

tdm_bigrams = TermDocumentMatrix(corp, control = ctrl2)
inspect(tdm_bigrams)
inspect(tdm_bigrams[c("research"),])
inspect(tdm_bigrams[c("research as"),])
Lenka Vraná
  • 1,686
  • 2
  • 19
  • 29
  • Hi Lenka, thank you. Yes the code i implemented is old somehow. when i tried your solution with the unigrams i got out of bound error. – Lian Ahmad May 05 '16 at 10:28
  • Did you try it with the same sentence that you provided as an example (`corpus2`)? You could get "out of bounds" error if the word "research" does not appear in your corpora. What output do you get when you run `inspect(tdm)`? – Lenka Vraná May 05 '16 at 10:34