1

I have a process that generates millions of small dataframes and save them to parquet in parallel.

all dataframes have the same columns and index information. and have the same number of rows (about 300).

as the dataframe is small, when they are saved into parquet files, the meta information is quite big in comparison with the data. as the meta information for each parquet file is basically the same, the disk space is wasted because the same meta are repeated millions of times.

is it possible to save one copy of meta information and other parquet files contains only the data ? when I need to read a dataframe, read the meta and the data from two different files?

some updates:

concating them into one big dataframe can save the disk space, but it's not friendly to do parallel processing of each small dataframe.

I also tried other format like feather.but it seems that feather does not store data as effciently as parquet. the file size is smaller but it's larger than parquet meta + parquet data

Lei Yu
  • 199
  • 9
  • How about just merging the small files into a larger dataframe then saving that? That's something I do a lot when generating modelling results across lots of machines. The larger dataframe will be smaller than the sum of the parts and much quicker to load than the small files. – John M. Oct 26 '22 at 08:29

1 Answers1

2

This is not possible at least using python pandas (fastparquet and pyarrow dont have any such feature).

I do see a parameter that disables statistics in footer.

However this will not save you a lot of disk space. Only few stats related info will not be written to the parquet metadata footer. enter image description here

is it possible to save one copy of meta information and other parquet files contains only the data ?

You are wanting to write multiple data files with only row groups without footer and a single metadata file that consists only footer. In this case none of those files will be a valid parquet file. This should be possible theoretically but no such known implementation exists. Checkout comments on this thread. Maybe reach out to parquet community on slack to find out if any such implementation exists.

My suggestions would be combine the dataframes somehow before writing to parquet format on disk. OR run a job at a later stage that will merge these files. Both these options are not efficient since you have a huge number of small df's/files.

You could write 300 rows per dataframe into a some kind of intermediate database as well. Later convert to parquet.

ns15
  • 5,604
  • 47
  • 51