Azure service bus trigger does not raise error message when calling a bad request to a URL and does not fail the job

Question

My service bus queue trigger will get triggered every time when a new file is landed in blob storage, and the trigger function will perform a post request to Airflow to trigger an airflow job, but doesn't matter if it is a good request with 200 or bad request 404, the trigger will mark as success automatically even when the airflow job does not get triggered, hence, we will lost that message and there is no way to retrieve it back.

I thought 2 solutions, firstly is catch error in Exception, secondly is manually failed the job if code != 200, but none of the solution works.

As we can see if the code, status = 404 and trigger function still marked succeeded. Where did I do wrong in my code? How could I make my logic work?

            try:
                result = requests.post(
                    f"http://localhost:8080/api/v1/dags/{dag_id}/dagRunss",
                    data = json.dumps(data),
                    headers = header,
                    auth = ("airflow", "airflow"))
                logging.info(result.content.decode('utf-8'))
                if result.status_code != 200:
                    raise ValueError('This is the exception you expect to handle')
            except Exception as err:
                logging.error(err)

Answer 1

I have modified your code with handling the response status code

try:
   result = requests.post(
                    f"http://localhost:8080/api/v1/dags/{dag_id}/dagRunss",
                    data = json.dumps(data),
                    headers = header,
                    auth = ("airflow", "airflow"))
        logging.info(result.content.decode('utf-8'))
    except Exception as err:
        logging.error(err)
    
    # Check the status code and trigger your airflow job
    if result.status_code == 200:
        # do your steps further like trigger your airflow job
        return "return your airflow job response "
    
    # If status_code != 200 do your other jobs or not trigger the airflow job
    else:
        logging.error(f' Status Code: {result.status_code} ')
        #here you will get other response status code like 40*,50*
        return "return your not 200 response and dont call the airflow job"
else:
   return " your airflow job / !=200 response"

if you got 404 response Status_Code, it will execute the code under first else: logging.error(f' Status Code: {result.status_code} '), return "return your not 200 response and dont call the airflow job".

Refer here for more information