-1

Need this info in the log as a print statement click for more info

Roshan_mobo
  • 3
  • 2
  • Do you need this info as part of a task in the dag at the nd ? Or query from outside with api its good? – ozs Jul 14 '22 at 14:04
  • Yes, both the way I need Like as a part of a task and Xcom value also. – Roshan_mobo Jul 15 '22 at 05:11
  • do you want the give dag duration in every task ? specific task that check the duration until this task ? part of the log or simply print it ? if you can add more details to your question it would be great – ozs Jul 15 '22 at 06:54
  • I have added an image in my question this is all the dag-related info I need can you pls provide this info in a task. – Roshan_mobo Jul 15 '22 at 07:53

1 Answers1

0

Assuming you need to get the duration of a DAG in a task in the DAG itself, then you need to put it as last task and need to understand there will be a little difference (cause the duration task is part of the DAG)

Here, an example of simple DAG that in the last task I calculate the duration and put it in the XCOM.

There is a bit difference also between XCOM and Airflow UI because rounding of the numbers.

enter image description here

enter image description here

from datetime import datetime, timedelta

from airflow import DAG
from airflow.decorators import task
from airflow.operators.python import get_current_context
from airflow.sensors.time_delta import TimeDeltaSensor
from airflow.utils import timezone

with DAG(
        dag_id="test_dag",
        start_date=datetime(2022, 1, 1),
        schedule_interval=None,
        render_template_as_native_obj=True,
        tags=["test"],
) as dag:

    @task
    def task1():
        print("task1")

    sleep_task = TimeDeltaSensor(
        task_id="sleep",
        delta=timedelta(seconds=3),
        mode='reschedule'
    )

    @task(multiple_outputs=True)
    def duration_task():
        context = get_current_context()
        dag_run = context["dag_run"]
        execution_date = dag_run.execution_date
        now = timezone.make_aware(datetime.utcnow())
        duration = now - execution_date
        return {
            "duration": str(duration),
            "start_time": str(dag_run.execution_date),
            "end_time": str(now)
        }

    (task1() >> sleep_task >> duration_task())
ozs
  • 3,051
  • 1
  • 10
  • 19