I would like to do incremental loads of my data based on partitioned data into SPICE, once I have my historical base. My data is partitioned by year/month/day and comes in real time. My spice query is using ATHENA as it comes from a GLUE crawler.
Here is something I have tried that DID not work
where (((current_date) <=cast('2022-11-26' as date))
and (( partition_0 = year(current_date) and partition_1 = month(current_date) )))
or
((current_date>cast('2022-11-26' as date))
and (( partition_0 = year(date_add('hour', -2, now())) and partition_1 = month(date_add('hour', -2, now())) and partition_2= day(date_add('hour', -2, now())) )))
Above, I am trying to grab every thing if it less than certain date, and only newer data after certain data. My incremental refresh is hourly. I am not sure if a manifest file is the way around this, but was unsure how to do that
