Unstacked DataFrame is too big, causing int32 overflow error when trying to use .pivot

Asked Apr 24 '20 at 13:18

Active Apr 24 '20 at 13:51

Viewed 451 times

When run, the following bit of code gives me the "Unstacked DataFrame is too big, causing int32 " error. movieID and userID are both uint16 and rating is uint8. How can I go around this?

from scipy.sparse import csr_matrix
# Switch ratings into movie features

df_movie_features = df_ratings.pivot(
    index='movieID',
    columns='userID',
    values='rating'
)

# Converts data of movie features to scipy sparse matrix
mat_movie_features = csr_matrix(df_movie_features.values)

edited Apr 24 '20 at 13:51

desertnaut

57,590
26
140
166

asked Apr 24 '20 at 13:18

user9325564

You will probably need to process you data in chunks. Perhaps dask. – It_is_Chris Apr 24 '20 at 13:22
Not a `scikit-learn` question - kindly do not spam irrelevant tags (removed). – desertnaut Apr 24 '20 at 13:51
Does this answer your question? [PANDAS: int32 overflow? Can't bulid a pivot table](https://stackoverflow.com/questions/56790261/pandas-int32-overflow-cant-bulid-a-pivot-table) – Muhammad Haseeb Khan Aug 09 '20 at 08:00
[This](https://stackoverflow.com/questions/56790261/pandas-int32-overflow-cant-bulid-a-pivot-table) helps. Use groupby instead of pivot – Max Sporer May 08 '21 at 16:16

Unstacked DataFrame is too big, causing int32 overflow error when trying to use .pivot

0 Answers0