Store data frame as parquet with mixed datatypes in one column (timestamp and string)

Question

I want to store a pandas data frame as Parquet file. But I got this error:

pyarrow.lib.ArrowTypeError: ("object of type <class 'str'> cannot be converted to int", 'Conversion failed for column foo with type object')

The column has mixed data types. I assume this is the problem. But how can I solve that?

#!/usr/bin/env python3
import pandas
df = pandas.DataFrame(
    data={
        'foo': [pandas.Timestamp('2022-06-01'), 'foobar']
        }
    )
print(df)

score 0 · Answer 1 · answered Jul 08 '22 at 14:56

Arrow only supports structured column data with a well defined type, so mixed types are not supported.

However you could use the pyarrow union type. But it's not very user friendly:

import pyarrow as pa
import pandas as pd

def series_to_union_array(series: pd.Series) -> pa.UnionArray:
    series_type = series.apply(type)
    types = series_type.unique().tolist()

    return pa.UnionArray.from_sparse(
        types=pa.array(series_type.apply(types.index), pa.int8()),
        children=[
            pa.array(series.where(series_type == dtype, None), None)
            for dtype in types
        ],
        field_names=[str(t) for t in types]
    )


df = pd.DataFrame(
    data={
        'foo': [pd.Timestamp('2022-06-01'), 'foobar']
        }
    )

table = pa.Table.from_arrays(
    [series_to_union_array(df['foo'])],
    names=['foo']
)

table['foo'].to_pylist()
>>> [datetime.datetime(2022, 6, 1, 0, 0), 'foobar']

Store data frame as parquet with mixed datatypes in one column (timestamp and string)

1 Answers1