0

I'm trying to load a Deltalake table in Python in a memory-bound environment (k8s Pod with memory limit) with Polars, I am getting an OOM exception when trying to do a scan_delta(...).head().collect().

I am unable to determine if predicate pushdown works and/or is implemented in Polars for Deltalake, or if my code is wrong.

martin8768
  • 565
  • 5
  • 20

1 Answers1

4

It does. However your query doesn't have a predicate (filter block), so there is no predicate pushdown.

Limits (the head call) are not yet pushed down in delta table. We can add that later.

ritchie46
  • 10,405
  • 1
  • 24
  • 43