0

This might be a dumb question, But is there any difference between manually specifying the partition columns in a parquet file, as opposed to loading it and then filtering them?

For Example: I have a parquet file that is partitioned by DATE. If I want the last 10 days, I can typically do the following: (assuming today is 2020-10-26)

df=spark.read.parquet("s3://bucket/path/file.parquet")\
.filter(col('DATE')>'2020-10-15')

Or I can use the S3 file system to load only the required partitions in the spark data frame like so:

inpath="s3://bucket/path/file.parquet/"
datepath=s3fs.S3FileSystem.ls(inpath)
dates=[]
for a in range(len(datepath)-10, len(datepath)):
    path="s3://" + datepath[a] + "/"
    dates=append(path)
df=spark.read.option("basePath", inpath).parquet(*dates)

The reason to do it the elaborate way in the 2nd method (in my thinking) was so that I did not have to load the entire parquet file with all the dates in memory and then filter them. I wanted to find out if my assumption is true.

Please advise. Thank You

thebluephantom
  • 16,458
  • 8
  • 40
  • 83
thentangler
  • 1,048
  • 2
  • 12
  • 38

1 Answers1

2

You can do both, but Spark predicate push-down has been around for a while, making your life easier. E.g. partition pruning and use of parquet stats such as min/max. See https://db-blog.web.cern.ch/blog/luca-canali/2017-06-diving-spark-and-parquet-workloads-example from 2017 and it also applies to pyspark.

Not everything van be pushed down, but other operators that can be pushed down are "<, <=, > , >=" as the link from 2017 shows. You can use .explain to check how the Optimizer does or does not apply predicate push-down.

thebluephantom
  • 16,458
  • 8
  • 40
  • 83
  • 1
    So not only can the I/O be significantly reduced by filtering on partitions, but also filtering on un-partitioned columns as long as the data types are valid and the data is not distributed evenly across. This saves me a bunch of lines of code without using `method 2`. Thank you for this information! – thentangler Oct 27 '20 at 17:52