0

Loading multiple files using glob patterns if we run it in a local filesystem, as it's written in the documentation.

However, if I try to load several files at once from the Azure Data lake Gen2, it only loads into the DataFrame the first file that matches the glob pattern.

from  polars.io import read_parquet

storage_options={'account_name': ACCOUNT_NAME, 'account_key': ACCOUNT_KEY}

parquet = read_parquet(
    "abfs://raw/files/file_202209*.parquet",
    storage_options=storage_options
)

Using the code from above only loads one file from the folder, but it doesn't not return an error or warning. enter image description here

I am using polars python version 0.14.15. Could it be a bug, or is loading multiple files only valid for local filesystems?

Javi Hernandez
  • 314
  • 8
  • 17

0 Answers0