-1

I have a task which is scheduled every few minutes.

I want to implement the logic where new task starts where previous successfully executed task left off.

More concretely I use this time intervals to than query the database and so I don't to miss some data between executions.

How this can be achieved ?

marknorkin
  • 3,904
  • 10
  • 46
  • 82

1 Answers1

0

Have a look at the documentation around macros. You can see that you have two variables that you can use in your sql files: {{ execution_date }} and {{ next_execution_date }} - you should use these to query the database for the time interval like this for example:

select * 
from 
  table
where
  timestamp_column >= {{ execution_date }} 
  and timestamp_column < {{ next_execution_date }}
Simon D
  • 5,730
  • 2
  • 17
  • 31
  • Of course it wouldn't, if you do a manual run and set the execution_date manually Airflow can't magically assume what you want for the next execution date... How would you do this in any other tool? – Simon D Oct 14 '18 at 19:06
  • I meant when there is a schedule interval set and you also sometimes start manually in between. – Programmer120 Oct 15 '18 at 06:25
  • I think you might have to update your question with an example since I don't really understand – Simon D Oct 15 '18 at 09:02