How to set a SLA in Airflow?

Question

I would like to set an SLA in a Sensor Operator. The documentation is not too clear about the use of it. So I did a test using the S3KeySensor operator which is looking for a file that does not exist. I set the SLA to 30 seconds, I was hoping to see the record after 30 seconds in the UI - in SLA misses - but it did not happen. What am I doing wrong?

inputsensor = S3KeySensor(
    task_id="check_for_files_in_s3",
    bucket_key="adp/backload/20136585/",
    wildcard_match=True,
    bucket_name="weblogs-raw",
    s3_conn_id="AWS_S3_CENTRAL",
    timeout=120,
    poke_interval=10,
    sla=timedelta(seconds=30),
    dag=dag,
)

inputsensor.set_downstream(next_step)

Instead of using sla, you have better fortune using execution_timeout setting the on_failure_callback accordely. — ozw1z5rd, Mar 18 '20 at 14:50

score 9 · Answer 1 · edited Oct 03 '21 at 03:44

9

"sla": timedelta(hours=2),

An example from Airflow Git repository: airflow/example_dags/tutorial.py#L54.

edited Oct 03 '21 at 03:44

Paul P

3,346
2
12
26

answered Jun 16 '17 at 23:54

Manish Ranjan

11,221
1
16
20

I find it strange that passing `SLA` in `default_args` works and passing it directly as param to `operator` doesn't work. Is this a known issue? – y2k-shubham Sep 05 '18 at 09:47

score 7 · Answer 2 · answered Jun 03 '20 at 09:24

According to the documentation SLA represents the timedelta after the schedule period is over. So if your schedule interval is '@daily' and sla=timedelta(hours=1) then Airflow will check for SLA misses at 1:00 AM which is when the schedule period is over plus one hour.

score 1 · Answer 3 · edited Oct 03 '21 at 07:39

1

In hours

"sla": timedelta(hours=2)

In minutes

"sla": timedelta(minutes=120)

edited Oct 03 '21 at 07:39

Paul P

3,346
2
12
26

answered May 19 '20 at 05:20

Ganesh

677
8
11

How to set a SLA in Airflow?

3 Answers3