Stack Overflow
Stack Overflow

Questions tagged [tfidfvectorizer]

Used in SKLearn to convert a collection of raw documents to a matrix of TF-IDF features.

Used in SKLearn to convert a collection of raw documents to a matrix of TF-IDF features.

http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html

410 questions
23
votes
4 answers

Use sklearn TfidfVectorizer with already tokenized inputs?

I have a list of tokenized sentences and would like to fit a tfidf Vectorizer. I tried the following: tokenized_list_of_sentences = [['this', 'is', 'one'], ['this', 'is', 'another']] def identity_tokenizer(text): return text tfidf =…
greenberet123
  • 1,351
  • 1
  • 12
  • 22
12
votes
1 answer

Confused with the return result of TfidfVectorizer.fit_transform

I wanted to learn more about NLP. I came across this piece of code. But I was confused about the outcome of TfidfVectorizer.fit_transform when the result is printed. I am familiar with what tfidf is but I could not understand what the numbers…
Huzo
  • 1,652
  • 1
  • 21
  • 52
11
votes
1 answer

How does TfidfVectorizer compute scores on test data

In scikit-learn TfidfVectorizer allows us to fit over training data, and later use the same vectorizer to transform over our test data. The output of the transformation over the train data is a matrix that represents a tf-idf score for each word for…
Yuval Cohen
  • 131
  • 1
  • 5
10
votes
1 answer

how to choose parameters in TfidfVectorizer in sklearn during unsupervised clustering

TfidfVectorizer provides an easy way to encode & transform texts into vectors. My question is how to choose the proper values for parameters such as min_df, max_features, smooth_idf, sublinear_tf? update: Maybe I should have put more details on the…
user6396
  • 1,832
  • 6
  • 23
  • 38
8
votes
0 answers

Converting TfidfVectorizer sparse matrix to dataframe or dense array results in memory error

My input is a pandas dataframe ("vector") with one column and 178885 rows holding strings with up to 600 words each. 0 this is an example text... 1 more examples... ... 178885 last example Name: vectortext, Length:…
cian
  • 191
  • 2
  • 11
7
votes
2 answers

TF-IDF vectorizer to extract ngrams

How can I use TF-IDF vectorizer from the scikit-learn library to extract unigrams and bigrams of tweets? I want to train a classifier with the output. This is the code from scikit-learn: from sklearn.feature_extraction.text import…
ECub Devs
  • 165
  • 3
  • 10
7
votes
4 answers

what is the difference between tfidf vectorizer and tfidf transformer

I know that the formula for tfidf vectorizer is Count of word/Total count * log(Number of documents / no.of documents where word is present) I saw there's tfidf transformer in the scikit learn and I just wanted to difference between them. I…
Jeeth
  • 2,226
  • 5
  • 24
  • 60
6
votes
1 answer

Reduce Dimension of word-vectors from TFIDFVectorizer / CountVectorizer

I want to use the TFIDFVectorizer (or CountVectorizer followed by TFIDFTransformer) to get a vector representation of my terms. That means, I want a vector for a term where the documents are the features. That's simply the transpose of a TF-IDF…
Highchiller
  • 194
  • 2
  • 11
6
votes
1 answer

Creating a TfidfVectorizer over a text column of huge pandas dataframe

I need to get matrix of TF-IDF features from the text stored in columns of a huge dataframe, loaded from a CSV file (which cannot fit in memory). I am trying to iterate over dataframe using chunks but it is returning generator objects which is not…
oldmonk
  • 691
  • 9
  • 16
6
votes
1 answer

When using the linear_kernel or the cosine_similarity for TfIdfVectorizer I get the error "Kernel died, restarting"

When using the linear_kernel or the cosine_similarity for TfIdfVectorizer, I get the error "Kernel died, restarting". I am running the scikit learn functions for TfID method Vectorizer and fit_transform on some text data like the example below, but…
ana
  • 61
  • 1
  • 4
5
votes
2 answers

Why does sklearn tf-idf vectorizer give the highest scores to stopwords?

I implemented Tf-idf with sklearn for each category of the Brown corpus in nltk library. There are 15 categories and for each of them the highest score is assigned to a stopword. The default parameter is use_idf=True, so I'm using idf. The corpus is…
khrystyna_s
  • 63
  • 3
5
votes
3 answers

Remove Stopwords in French AND English in TfidfVectorizer

I am trying to remove stopwords in French and English in TfidfVectorizer. So far, I've only managed to remove stopwords from the English language. When I try to enter the French language for the stop_words, I get an error message that says it's not…
OnThaRise
  • 117
  • 1
  • 1
  • 9
5
votes
3 answers

Find top n terms with highest TF-IDF score per class

Let's suppose that I have a dataframe with two columns in pandas which resembles the following one: text label 0 This restaurant was amazing Positive 1 The food was served cold Negative 2 …
Outcast
  • 4,967
  • 5
  • 44
  • 99
5
votes
1 answer

Combining TF-IDF with pre-trained Word embeddings

I have a list of website meta-description (128k descriptions; each with avg. 20-30 words), and am trying to build a similarity ranker (as in: show me the 5 most similar sites to this site meta description) It worked AMAZINGLY well with TF-IDF uni-…
benjo121212
  • 75
  • 1
  • 6
5
votes
1 answer

How to Select Top 1000 words using TF-IDF Vector?

I have a Documents with 5000 reviews. I applied tf-idf on that document. Here sample_data contains 5000 reviews. I am applying tf-idf vectorizer on the sample_data with one gram range. Now I want to get the top 1000 words from the sample_data which…
merkle
  • 1,585
  • 4
  • 18
  • 33
1
2 3
…
27 28