How to override the parameters of default args in a custom operator?

Question

Hello I have a requirement where I have a custom operators in a dag and I want to override the parameter in one of the custom operator.But in custom operator @apply_defaults has been used.In this case if I override the value of a parameter of default args in custom parameter which parameter will be used.Will it be default args parameter or the one that we have overridden

Below is the example:

from airflow.models.baseoperator import BaseOperator
from airflow.utils.decorators import apply_defaults
from hooks.my_hook import MyHook

from airflow import DAG
from datetime import datetime, timedelta
from operators.my_operator import MyOperator
from sensors.my_sensor import MySensor


class MyOperator(BaseOperator):

    @apply_defaults
    def __init__(self,
                 my_field,
                 *args,
                 **kwargs):
        super(MyOperator, self).__init__(*args, **kwargs)
        self.my_field = my_field

    def execute(self, context):
        hook = MyHook('my_conn')
        hook.my_method()






default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2018, 1, 1),
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}


with DAG('example_dag',
         max_active_runs=3,
         schedule_interval='@once',
         default_args=default_args) as dag:

    sens = MySensor(
        task_id='taskA'
    )

    op = MyOperator(
        task_id='taskB',
        my_field='some text',
        retries=2
    )

    sens >> op

So in the above case which retries will be used.Weather it will be default_args retries or the one that I have overridden in custom operator.

I am bit confused about the functionality of apply_defaults here.I have a doubt that @apply_defaults takes default_args or the overidden retries

score 0 · Answer 1 · answered Apr 28 '22 at 18:22

apply_defaults is decorator that looks for an argument named "default_args", and fills the unspecified arguments from it.

So in your case retries=2 that you specify in MyOperator will supersede any other definition.

Noting that since Airflow 2.1 there is no reason to use @apply_defaults anymore. See release notes.

How to override the parameters of default args in a custom operator?

1 Answers1