Airflow Error callback "on_failure_callback" is not executing all the lines in the function

Question

I have a problem in the usage of the on_failure_callback. I have defined my error callback function to perform 2 "http post" requests and I have added a logging.error( ) message between the two. I notice that only one is getting executed. Is there any delay or some thing that I am missing here? please help.

def custom_failure_function(context):
        logging.error("These task instances ahhh")
        to_json= json.loads(t_teams)
        var1= json.dumps(to_json)
        print(var1)
        r = requests.post('https://myteamschannel/teams', data=var1,verify=False)
        logging.error("hello")
        runID='OPERATION_CONTEXT .OCV8.TEST2 alarm_object 193'
        headers = {'Content-Type':'text/xml'}
        alarmRequest='<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:oper=\"http://172.19.146.147:7180/TeMIP_WS/services/OPERATION_CONTEXT-alarm_object\"><soapenv:Header xmlns:wsa=\"http://www.w3.org/2005/08/addressing\"><wsu:Timestamp xmlns:wsu=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\"><wsu:Created>2014-05-22T11:57:38.267Z</wsu:Created><wsu:Expires>2014-05-22T12:02:38.000Z</wsu:Expires></wsu:Timestamp><wsse:Security xmlns:wsse=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd\" soapenv:mustUnderstand=\"1\"><wsse:UsernameToken xmlns:wsu=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd\"><wsse:Username>girws</wsse:Username><wsse:Password Type=\"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText\">Temip</wsse:Password></wsse:UsernameToken></wsse:Security></soapenv:Header> <soapenv:Body> <oper:Set_Request xmlns:oper=\"http://172.19.146.147:7180/TeMIP_WS/services/OPERATION_CONTEXT-alarm_object\"><EntitySpec><Natural> '+ runID +  '</Natural></EntitySpec><Arguments> <Attribute_Values><Filtering_Type>' + 'AUTOFAIL' + '</Filtering_Type></Attribute_Values></Arguments></oper:Set_Request> </soapenv:Body> </soapenv:Envelope>'
        r = requests.post('http://myerrorappli:7180/TeMIP_WS/services/OPERATION_CONTEXT-alarm_object', header=headers, data=alarmRequest,verify=True)
        logging.error ("FAILED TASK")
        logging.error("============================================")

The logs of my airflow are below. Its stopping at the "hello" message and not printing "FAILED TASK".

*** Reading local file: /data/airflow//logs/MOE_TEST_DAG/TeamsTest/2021-10-02T08:24:14.821970+00:00/3.log

[2021-10-02 10:24:36,535] {MOE_TEST.py:132} ERROR - These task instances ahhh
[2021-10-02 10:24:36,987] {MOE_TEST.py:138} ERROR - hello

try to set timeout for requests.post https://stackoverflow.com/questions/54619868/http-requests-post-timeout/54619967 — Elad Kalif, Oct 02 '21 at 12:23

score 1 · Answer 1 · answered Oct 02 '21 at 21:06

From your description it's more likely that there is an issue with requests.post() try to add timeout to the request:

def custom_failure_function(context):
    ...
    try:
        r = requests.post('http://myerrorappli:7180/TeMIP_WS/services/OPERATION_CONTEXT-alarm_object', header=headers,
                      data=alarmRequest, verify=True, timeout=5)
    except requests.Timeout:
        logging.error("request timeout")
    except requests.ConnectionError:
        logging.error("request connection error")
        
    logging.error("FAILED TASK")
    logging.error("============================================")

Airflow Error callback "on_failure_callback" is not executing all the lines in the function

1 Answers1