2

I create a DAG to do monitoring of other applications, when the DAG does not fulfill the monitoring task, it sends an alert email. My DAG has a ten minutes interval scheduler, that is, if the task fails, it sends emails every ten minutes, but what I want to achieve is to send me the first email at ten minutes, the second email at fifteen minutes, the third email at thirty minutes and continue as long as the condition of the previous task is failed, if the task is successful continue monitoring every ten minutes.


current_date = datetime.now()
default_args = {
    'owner': 'my_dag',
    'start_date':  datetime(2021, 3, 25, current_date.hour),
    'timeout': 300,  --> seconds
    'poke_interval': 20, --> seconds
    'exponential_backoff': True
    }


with DAG(
    dag_id="Health_Check",
    schedule_interval="*/10 8-19 * * *", # At every 10th minutes past every hour from 8 through 20.
    tags=["Health_Check"],
    default_args=default_args,
    catchup=False
        ) as dag

    do_request = CustomSensor(
        task_id='do_request',
        endpoint='/Url',
        method='GET',
        http_conn_id='conn_api'
        )

I try implement exponential retracement at the DAG level because in the tasks I use exponential_backoff this attribute of the sensor operator so that when it tries to generate a request to a URL it tries it the first time if it fails it tries 20 seconds later if it fails it tries 40s later if it fails 80s after and continue in this way until it falls on time out.

I would like to implement the same, but at the DAG level, any idea??

John_B
  • 41
  • 8
  • I don't understand why you want this on DAG level. Airflow schedule tasks not DAGs. Please add example where according to you it make sense to have it on DAG level rather than task level. – Elad Kalif Apr 17 '21 at 10:38
  • Let me explain, the DAG has an interval of ten minutes, it monitors several applications, if any application fails, the entire DAG fails and sends an email that shows the error information, but I want the intervals to be dynamic so that I don't get this showing the same error every 10 min. If it fails send an increasingly extended email, obviously until the error is resolved – John_B Apr 18 '21 at 01:43
  • Please post your full dag code including the email alert body. This can be solved. – Elad Kalif Apr 18 '21 at 03:10
  • 8
    Did your recently [deleted question](https://stackoverflow.com/questions/67233828/airflow-task-decorator-sample) was actually meant to be an answer here? It is completely fine (and even encouraged) to answer your own questions if you found a solution that can help others in the future – Tomerikoo Apr 23 '21 at 21:06
  • 3
    No, it was not an answer to this question, but it was an answer that I thought useful for several people who use Airflow. I'm going to edit the previous post correctly, thank you very much for the feedback,greetings. – John_B Apr 23 '21 at 21:51

0 Answers0