How can i read all parquet files from all subfolder

Question

folder achitecture:

dir1
 |-dir1-1
 |    |-file1.parquet
 |-dir1-2
     |-file2.parquet

how can i read all parquet files, i try to use

sparkSession.read.format("parquet")
      .option("recursiveFileLookup", "true")
      .load("dir1path")

but it shows error: unable to infer schema for parquet. it must be specified manully

Does this answer your question? [Read all files in a nested folder in Spark](https://stackoverflow.com/questions/32233575/read-all-files-in-a-nested-folder-in-spark) — Jarrod Baker, Oct 07 '21 at 07:10

kailing · Answer 1 · 2021-10-07T00:11:30.673

0

my current solution is: sparkSession.read.parquet("dir1/*")

edited Oct 07 '21 at 00:11

answered Oct 07 '21 at 00:06

kailing

1 Answers1