1

I am trying to read data from a large parquet file of 30G. My memory do not support default reading with fastparquet in python, so I do not know what I should do to lower the memory usage of the reading process.

Kehan Chen
  • 11
  • 1

1 Answers1

1

You can use pyarrow's iter_batches to read back chunks of rows incrementally.

Micah Kornfield
  • 1,325
  • 5
  • 10