Converting TfidfVectorizer sparse matrix to dataframe or dense array results in memory error

Question

My input is a pandas dataframe ("vector") with one column and 178885 rows holding strings with up to 600 words each.

0         this is an example text...
1         more examples...
          ...
178885    last example
Name: vectortext, Length: 178886, dtype: object

I'm doing feature extraction (unigrams) using the TfidfVectorizer:

vectorizer_uni = TfidfVectorizer(ngram_range=(1,1), use_idf=True, analyzer="word", stop_words=stop)
X = vectorizer_uni.fit_transform(vector).toarray()
X = pd.DataFrame(X, columns=vectorizer_uni.get_feature_names()) #map grams 
k = len(X.columns) #number of features

Unfortunately I'm receiving a Memory Error as below. I'm using the 64bit version of python 3.6 with 16GB RAM on my windows 10 machine. I've red alot about python generators etc. but I can't figure out how to solve this problem without limiting the number of features (which is not really an option). Any ideas how to solve this? Could I somehow split my dataframe before?

Traceback:

---------------------------------------------------------------------------
 MemoryError                               Traceback (most recent call last)
       <ipython-input-88-15b6091ceec7> in <module>()
       1 vectorizer_uni = TfidfVectorizer(ngram_range=(1,1), use_idf=True, analyzer="word", stop_words=stop)
 ----> 2 X = vectorizer_uni.fit_transform(vector).toarray()
       3 X = pd.DataFrame(X, columns=vectorizer_uni.get_feature_names()) #map grams
       4 k = len(X.columns) # number of features

 C:\Programme\Anaconda3\lib\site-packages\scipy\sparse\compressed.py in toarray(self, order, out)
       962     def toarray(self, order=None, out=None):
       963         """See the docstring for `spmatrix.toarray`."""
   --> 964         return self.tocoo(copy=False).toarray(order=order, out=out)
       965 
       966     ##############################################################

 C:\Programme\Anaconda3\lib\site-packages\scipy\sparse\coo.py in toarray(self, order, out)
       250     def toarray(self, order=None, out=None):
       251         """See the docstring for `spmatrix.toarray`."""
   --> 252         B = self._process_toarray_args(order, out)
       253         fortran = int(B.flags.f_contiguous)
       254         if not fortran and not B.flags.c_contiguous:

 C:\Programme\Anaconda3\lib\site-packages\scipy\sparse\base.py in _process_toarray_args(self, order, out)
       1037             return out
       1038         else:
    -> 1039             return np.zeros(self.shape, dtype=self.dtype, order=order)
       1040 
       1041     def __numpy_ufunc__(self, func, method, pos, inputs, **kwargs):

 MemoryError:

@MikePalmice It's probably happening at `X = vectorizer_uni.fit_transform(vector).toarray()` because he is converting a sparse matrix to an array. — Scratch'N'Purr, Feb 20 '18 at 13:48
Is it necessary to convert X into a dataframe? If you work with the sparse matrix directly there will be no problem — Artur Lacerda, Feb 20 '18 at 13:53
@ArturLacerda you were right! Total misconception... Thanks for pointing this out, I'm working with a sparse matrix now, which works perfectly fine! Thank you! — cian, Feb 21 '18 at 10:29
Hey @cian could you please tell us how to handle this error, I am having a dataframe with around 200k records, how to get it's tfidf vectorized array — Mousam Singh, Aug 28 '19 at 10:21
After removing toarray() method we have sparse matrix of features but how can we convert this into array to work on it. — Mousam Singh, Aug 28 '19 at 10:29

Converting TfidfVectorizer sparse matrix to dataframe or dense array results in memory error

0 Answers0

Linked