Why index name always appears in the parquet file created with pandas?

Question

I am trying to create a parquet using pandas dataframe, and even though I delete the index of the file, it is still appearing when I am re-reading the parquet file. Can anyone help me with this? I want index.name to be set as None.

>>> df = pd.DataFrame({'key': 1}, index=[0])
>>> df
  key
0    1
>>> df.to_parquet('test.parquet')
>>> df = pd.read_parquet('test.parquet')
>>> df
     key
index     
0        1
>>> del df.index.name
>>> df
     key
0    1
>>> df.to_parquet('test.parquet')
>>> df = pd.read_parquet('test.parquet')
>>> df
     key
index     
0        1

Is this using fastparquet or pyarrow as the Parquet backend? — Uwe L. Korn, Aug 16 '18 at 08:24

score 4 · Accepted Answer · answered Aug 16 '18 at 10:24

It works as expected using pyarrow:

>>> df = pd.DataFrame({'key': 1}, index=[0])
>>> df.to_parquet('test.parquet', engine='fastparquet')
>>> df = pd.read_parquet('test.parquet')
>>> del df.index.name
>>> df
   key
0    1
>>> df.to_parquet('test.parquet', engine='fastparquet')
>>> df = pd.read_parquet('test.parquet')
>>> df
       key
index     
0        1 ---> INDEX NAME APPEARS EVEN AFTER DELETING USING fastparquet
>>> del df.index.name
>>> df.to_parquet('test.parquet', engine='pyarrow')
>>> df = pd.read_parquet('test.parquet')
>>> df
   key
0    1 --> INDEX NAME IS NONE WHEN CONVERSION IS DONE WITH pyarrow

score 3 · Answer 2 · answered Jan 22 '20 at 10:28

Hey this works with pyarrow with the following

df = pd.DataFrame({'key': 1}, index=[0])
df.to_parquet('test.parquet', engine='pyarrow', index=False)
df = pd.read_parquet('test.parquet', engine='pyarrow')
df.head()

As @alexopoulos7 mentioned in the to_parquet documentation it states you can use the "index" argument as a parameter. It seems to work, perhaps because I'm explicitly stating the engine='pyarrow'

score 2 · Answer 3 · answered Oct 23 '19 at 10:09

I have been playing with both libraries pyarrow and fastparquet, trying to write a parquet file without preserving indexes since I need those data to be read from redshift as an external table.

For me what it worked was for library fastparquet

df.to_parquet(destination_file, engine='fastparquet', compression='gzip', write_index=False)

If you try to follow the to_parquet official documentation you will see that it mentions parameter "index" but this throws an error if this argument does not exist in the used engine. Currently, I have found that only fastparquet has such an option and in named "write_index"

Why index name always appears in the parquet file created with pandas?

3 Answers3