I have a DAG that fetches stock data after market hours 16:30 EST. I need to fetch the same date as the date parameter in my task to fetch 1 year old stock data. I initially used {{ds}} and found out {{ds}} only passed the interval before the run date by design. (ie, a DAG I schedule to run on 2022-10-11 with daily @ 16:30 EST interval will have a {{ds}} == '2022-10-10'. I found {{ data_interval_end }} may do the trick but get the following error since {{data_interval_end}} is in datetime format. Is there way I could format to "YYYY-MM-DD"" within the Python Operator?
ValueError: unconverted data remains: T20:45:00+00:00
execution_date = "{{ data_interval_end }}"
with DAG("DAG_SAMPLE",
start_date=pendulum.datetime(2021, 1 ,1, tz="US/Pacific"),
schedule_interval='45 13 * * 1-5', #At 01:45 PM, Monday through Friday
catchup=False) as dag:
scanning_swing_pattern_task = PythonOperator(
task_id='scanning_swing_pattern_task',
python_callable=scanning_swing_pattern.main,
op_kwargs={'run_date': execution_date}
)