Python 3: Feather cannot serialize numpy arrays

Asked Sep 25 '17 at 05:23

Active Oct 25 '17 at 12:06

Viewed 2,050 times

I'm trying to save my pandas dataframe to feather, but I'm having some problems with values column (It's a numpy array)... This is my df:

  name   values
1 so     [1, 3, 5, 6] 
2 ed     [54968, 9, 02]
3 jon    [5, 34349, 02, ..., 345546]

When I try to save it:

with NamedTemporaryFile("w") as output_file: 
    df.to_feather(output_file.name)
*** ValueError: cannot serialize column 1 named values with dtype mixed

I also tried to save values with pickle, without success...

import pickle
df['values'] = pickle.dumps(out_df['values'], protocol=0)
df.to_feather(output_file.name)
*** ValueError: cannot serialize column 1 named values with dtype bytes

I don't want to use csv. What can I do?

Edit1

sample column cells are representing samples of measurements. This means that the length of samples can go from hundreds to thousands of values. Thus the length of the numpy arrays is very variable.

df.dtypes values object name object

edited Oct 25 '17 at 12:06

kgf3JfUtW

13,702
10
57
80

asked Sep 25 '17 at 05:23

enriquetaso

Only two columns? – juanpa.arrivillaga Sep 25 '17 at 05:28
Yes.. only two columns. – enriquetaso Sep 25 '17 at 05:29
what's `df.dtypes` – juanpa.arrivillaga Sep 25 '17 at 05:34
Consider expanding your columns and then saving them. Using columns of lists kills all vectorisation benefits pandas provides. – cs95 Sep 25 '17 at 06:02
I updated the description. – enriquetaso Sep 26 '17 at 23:36
@enriquetaso how did you get around this? I'm having the same issue... – AJMA Jan 30 '19 at 20:31
When I had a column of ragged float arrays, I was not able to use `pd.DataFrame.to_feather`, just `pd.DataFrame.to_pickle`. Can you cast your arrays to a different dtype? Maybe something like `df.values = df.values.apply(lambda _: _.astype(np.int64))`. (swifter can help apply faster) – michen00 Nov 10 '21 at 07:25

Python 3: Feather cannot serialize numpy arrays

0 Answers0