Reading spark partitioned data from directories

Question

My data is partitioned as Year,month,day in s3 Bucket. I have a requirement to read last six months of data everyday.I am using below code to read the data but it is selecting negative value in months. is there a way to read the correct data for last six months?

from datetime import datetime
d = datetime.now().day
m = datetime.now().month
y = datetime.now().year
df2=spark.read.format("parquet") \
  .option("header","true").option("inferSchema","true") \
  .load("rawdata/data/year={2021,2022}/month={m-6,m}/*")

I think you can have a list of strings inside `load()`, i.e., `load(['rawdata/../year=2021/month=12', 'rawdata/../year=2022/month=1'])` — Ala Tarighati, Jan 10 '22 at 11:36
@AlaTarighati as per your code it reads only two months data. — code_bug, Jan 10 '22 at 13:30
Yes, I wanted to give you an idea on how this can be done. I'll try to write an answer then. — Ala Tarighati, Jan 10 '22 at 14:18
Are you having multi-level partitioning? it seems so. Not advised normally. — thebluephantom, Jan 10 '22 at 15:58
Yes it is.any performance issues with this type of partitioning? — code_bug, Jan 10 '22 at 18:20

score 1 · Accepted Answer · answered Jan 10 '22 at 14:33

You can use a list of addresses (strings) as your .load() argument. First you can create the list for six months backward (from today):

from datetime import date
from dateutil.relativedelta import relativedelta

y_m_list = [((date.today()+relativedelta(months=-i)).year, (date.today()+relativedelta(months=-i)).month)  for i in range(0,6)]

y_m_list

Output:

[(2022, 1), (2021, 12), (2021, 11), (2021, 10), (2021, 9), (2021, 8)]

Then create the argument as

.load([f"rawdata/data/year={x}/month={y}" for x,y in y_m_list])

Reading spark partitioned data from directories

1 Answers1