Preserve index when loading pyarrow parquet from pandas DataFrame

Question

I need to convert a dict with dict values to parquet, I have data that look like this:

{"KEY":{"2018-12-06":250.0,"2018-12-07":234.0}}

I'm converting to pandas dataframe and then writing to pyarrow table:

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

data = {"KEY":{"2018-12-06":250.0,"2018-12-07":234.0}}
df = pd.DataFrame.from_dict(data, orient='index')
table = pa.Table.from_pandas(df, preserve_index=False)
pq.write_table(table, 'file.parquet', flavor='spark')

I end up with data, that only have dates and values, but without the key of the dict.:

{"2018-12-06":250.0,"2018-12-07":234.0}

What I need is to also have the key of the data:

{"KEY": {"2018-12-06":250.0,"2018-12-07":234.0}}

Why did you set `preserve_index=False` if you _wanted_ to preserve the index? — cs95, Dec 05 '18 at 21:11

score 3 · Accepted Answer · edited Dec 05 '18 at 21:25

3

If you wanted to preserve the index, then you should've specified as such; set preserve_index=True:

table = pa.Table.from_pandas(df, preserve_index=True)

pq.write_table(table, 'file.parquet', flavor='spark')
pq.read_table('file.parquet').to_pandas()  # Index is preserved.

     2018-12-06  2018-12-07
KEY       250.0       234.0

edited Dec 05 '18 at 21:25

unixeo

1,116
1
15
28

answered Dec 05 '18 at 21:14

cs95

379,657
97
704
746

score 1 · Answer 2 · answered Jun 24 '21 at 23:11

I am observing a related but separate issue where the the frequency type of DateTimeIndex is not preserved in round trip from pandas to table.

For example:

    >>> import pandas as pd
    >>> import pyarrow as pa
    >>> from collections import OrderedDict
    >>>
    >>>
    >>> pd.__version__
    '1.1.5'
    >>>
    >>> pa.__version__
    '4.0.1'
    >>>
    >>> dates = pd.date_range(start='2016-04-01', periods=4, name='DATE')
    >>> dict_data = OrderedDict()
    >>> dict_data['A'] = list('AABB')
    >>> dict_data['B'] = list('abab')
    >>> dict_data['C'] = list('wxyz')
    >>> dict_data['D'] = range(0, 4)
    >>> df = pd.DataFrame.from_dict(dict_data)
    >>> df = df.set_index(dates)
    >>>
    >>> df.index
    DatetimeIndex(['2016-04-01', '2016-04-02', '2016-04-03', '2016-04-04'], dtype='datetime64[ns]', name='DATE', freq='D')
    >>>
    >>> table = pa.Table.from_pandas(df, preserve_index=True)
    >>> df2 = table.to_pandas()
    >>> df2.index
    DatetimeIndex(['2016-04-01', '2016-04-02', '2016-04-03', '2016-04-04'], dtype='datetime64[ns]', name='DATE', freq=None)

There is a pandas issue for this: https://github.com/pandas-dev/pandas/issues/41543 — kadee, Mar 01 '23 at 14:35

Preserve index when loading pyarrow parquet from pandas DataFrame

2 Answers2