2

I have a large Parquet file with just a few columns, and I'd like apply a function to each row of the Paruqet file. What is an efficient way of doing that? By efficient, I mean 2 things.

  1. The code runs fast.
  2. The code doesn't load the while file into memory but rather process it a few rows at a time.
Chayim Friedman
  • 47,971
  • 5
  • 48
  • 77
Benjamin Du
  • 1,391
  • 1
  • 17
  • 25
  • 1
    Efficient and row wise access do not match well in polars. We have columnar memory. Maybe you can ask a more conrete question and ask how you can do achieve that goal efficient. – ritchie46 May 26 '22 at 18:18
  • The [parquet](https://crates.io/crates/parquet) crate does allow row-wise access to Parquet files. I have been using it in the past and is wondering whether I can totally replace it with polars. – Benjamin Du May 27 '22 at 01:47

0 Answers0