Following is my code:
sklearn_tfidf = TfidfVectorizer(ngram_range= (3,3),stop_words=stopwordslist, norm='l2',min_df=0, use_idf=True, smooth_idf=False, sublinear_tf=True)
sklearn_representation = sklearn_tfidf.fit_transform(documents)
It generates tri gram by removing all the stopwords.
What I want it to allow those TRIGRAM what have stopword in their middle ( not in start and end)
Is there processor needs to be written for this. Need suggestions.
Yes, you need to supply your own analyzer function which will convert the documents to the features as per your requirements.
According to the documentation:
analyzer : string, {‘word’, ‘char’, ‘char_wb’} or callable
.... .... If a callable is passed it is used to extract the sequence of features out of the raw, unprocessed input.
In that custom callable you need to take care of first splitting the sentence into different parts, removing special chars like comma, braces, symbols etc, convert them to lower case, then convert them to n_grams.
The default implementation works on a single sentences in the following order:
max_df or lower than min_df.You need to handle all this if you want to pass a custom callable to the analyzer param in the TfidfVectorizer.
OR
You can extend the TfidfVectorizer class and only override the last 2 steps. Something like this:
from sklearn.feature_extraction.text import TfidfVectorizer
class NewTfidfVectorizer(TfidfVectorizer):
def _word_ngrams(self, tokens, stop_words=None):
# First get tokens without stop words
tokens = super(TfidfVectorizer, self)._word_ngrams(tokens, None)
if stop_words is not None:
new_tokens=[]
for token in tokens:
split_words = token.split(' ')
# Only check the first and last word for stop words
if split_words[0] not in stop_words and split_words[-1] not in stop_words:
new_tokens.append(token)
return new_tokens
return tokens
Then, use it like:
vectorizer = NewTfidfVectorizer(stop_words='english', ngram_range=(3,3))
vectorizer.fit(data)
