How to save simulation timestep data in a parquet file and then open a new file after every x amount of steps to save memory? (Python)

Asked Feb 03 '23 at 08:59

Active Feb 03 '23 at 08:59

Viewed 26 times

I have a cantera simulation which sends each timestep of a chemical reaction into a parquet file where a lot of species from the DataFrame are used. The problem is that it takes too long for the simulation to finish.

Is it possible to write the simulation steps in one parquet file and then let's say, after 5000 steps (automatically) open a new parquet file and continue saving the timesteps there.

I have tried using array's and repartition but no parquet file is being written at all.

asked Feb 03 '23 at 08:59

Dominator10

If you have enough memory, you're going to be better off accumulating all the results in a dataframe or array in memory and writing to disk once at the end. – darthbith Mar 13 '23 at 01:31

How to save simulation timestep data in a parquet file and then open a new file after every x amount of steps to save memory? (Python)

0 Answers0