How to build a gensim dictionary that includes bigrams?

Question

I'm trying to build a Tf-Idf model that can score bigrams as well as unigrams using gensim. To do this, I build a gensim dictionary and then use that dictionary to create bag-of-word representations of the corpus that I use to build the model.

The step to build the dictionary looks like this:

dict = gensim.corpora.Dictionary(tokens)

where token is a list of unigrams and bigrams like this:

[('restore',),
 ('diversification',),
 ('made',),
 ('transport',),
 ('The',),
 ('grass',),
 ('But',),
 ('distinguished', 'newspaper'),
 ('came', 'well'),
 ('produced',),
 ('car',),
 ('decided',),
 ('sudden', 'movement'),
 ('looking', 'glasses'),
 ('shapes', 'replaced'),
 ('beauties',),
 ('put',),
 ('college', 'days'),
 ('January',),
 ('sometimes', 'gives')]

However, when I provide a list such as this to gensim.corpora.Dictionary(), the algorithm reduces all tokens to bigrams, e.g.:

test = gensim.corpora.Dictionary([(('happy', 'dog'))])
[test[id] for id in test]
=> ['dog', 'happy']

Is there a way to generate a dictionary with gensim that includes bigrams?

qaiser · Answer 1 · 2021-07-20T06:07:39.663

from gensim.models import Phrases
from gensim.models.phrases import Phraser
from gensim import models



docs = ['new york is is united states', 'new york is most populated city in the world','i love to stay in new york']

token_ = [doc.split(" ") for doc in docs]
bigram = Phrases(token_, min_count=1, threshold=2,delimiter=b' ')


bigram_phraser = Phraser(bigram)

bigram_token = []
for sent in token_:
    bigram_token.append(bigram_phraser[sent])

output will be : [['new york', 'is', 'is', 'united', 'states'],['new york', 'is', 'most', 'populated', 'city', 'in', 'the', 'world'],['i', 'love', 'to', 'stay', 'in', 'new york']]

#now you can make dictionary of bigram token 
dict_ = gensim.corpora.Dictionary(bigram_token)

print(dict_.token2id)
#Convert the word into vector, and now you can use tfidf model from gensim 
corpus = [dict_.doc2bow(text) for text in bigram_token]

tfidf_model = models.TfidfModel(corpus)

being nit-picky but you should really rename the variable 'dict' to something else as 'dict()' is already the name of a built-in constructor — Silent-J, Jul 20 '21 at 02:07

fbparis · Answer 2 · 2019-01-28T05:47:48.833

You have to "phrase" your corpus to detect bigrams before creating your dictionary.

I would suggest you also stem or lemmatize it before feeding the dictionary, here is an example with nltk stemmer function:

import re
from gensim.models.phrases import Phrases, Phraser
from gensim.corpora.dictionary import Dictionary
from gensim.models import TfidfModel
from nltk.stem.snowball import SnowballStemmer as Stemmer

stemmer = Stemmer("YOUR_LANG") # see nltk.stem.snowball doc

stopWords = {"YOUR_STOPWORDS_FOR_LANG"} # as a set

docs = ["LIST_OF_STR"]

def tokenize(text):
    """
    return list of str from a str
    """
    # keep lowercase alphanums and "-" but not "_"
    return [w for w in re.split(r"_+|[^\w-]+", text.lower()) if w not in stopWords]

docs = [tokenize(doc) for doc in docs]
phrases = Phrases(docs)
bigrams = Phraser(phrases)
corpus = [[stemmer.stem(w) for w in bigrams[doc]] for doc in docs]
dictionary = Dictionary(corpus)
# and here is your tfidf model:
tfidf = TfidfModel(dictionary=dictionary, normalize=True)

How to build a gensim dictionary that includes bigrams?

2 Answers2