Amazon Managed Airflow 2.2.2 Fails To Run Tasks From The CLI With DeprecationWarning

Question

Context

I am upgrading our system from Amazon Managed Airflow 2.0.2 to 2.2.2. I am running into a situation where I can run DAGs in the UI but if I try to run them from the API I'm hitting

/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:357 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.

I'm trying to figure out what could be causing this problem or even what my next steps to debug this issue are.

DAGs Running In UI

The dag in this example is failing but that's not what I'm focused on currently. I can't even get the dags to start from the CLI.

Example MWAA CLI Call

% curl --request POST "https://$WEB_SERVER_HOSTNAME/aws_mwaa/cli" -sS --header "Authorization: Bearer $CLI_TOKEN" --header "Content-Type: text/plain" --data-raw "$OPERATION"
{"stderr":"L3Vzci9sb2NhbC9haXJmbG93Ly5sb2NhbC9saWIvcHl0aG9uMy43L3NpdGUtcGFja2FnZXMvYWlyZmxvdy9jb25maWd1cmF0aW9uLnB5OjM1NyBEZXByZWNhdGlvbldhcm5pbmc6IFRoZSBkZWZhdWx0X3F1ZXVlIG9wdGlvbiBpbiBbY2VsZXJ5XSBoYXMgYmVlbiBtb3ZlZCB0byB0aGUgZGVmYXVsdF9xdWV1ZSBvcHRpb24gaW4gW29wZXJhdG9yc10gLSB0aGUgb2xkIHNldHRpbmcgaGFzIGJlZW4gdXNlZCwgYnV0IHBsZWFzZSB1cGRhdGUgeW91ciBjb25maWcuCi91c3IvbG9jYWwvYWlyZmxvdy8ubG9jYWwvbGliL3B5dGhvbjMuNy9zaXRlLXBhY2thZ2VzL2FpcmZsb3cvY29uZmlndXJhdGlvbi5weTozNTcgRGVwcmVjYXRpb25XYXJuaW5nOiBUaGUgZGVmYXVsdF9xdWV1ZSBvcHRpb24gaW4gW2NlbGVyeV0gaGFzIGJlZW4gbW92ZWQgdG8gdGhlIGRlZmF1bHRfcXVldWUgb3B0aW9uIGluIFtvcGVyYXRvcnNdIC0gdGhlIG9sZCBzZXR0aW5nIGhhcyBiZWVuIHVzZWQsIGJ1dCBwbGVhc2UgdXBkYXRlIHlvdXIgY29uZmlnLgo=","stdout":"ZW52X2Rldl9tYWluCg=="}
% echo "L3Vzci9sb2NhbC9haXJmbG93Ly5sb2NhbC9saWIvcHl0aG9uMy43L3NpdGUtcGFja2FnZXMvYWlyZmxvdy9jb25maWd1cmF0aW9uLnB5OjM1NyBEZXByZWNhdGlvbldhcm5pbmc6IFRoZSBkZWZhdWx0X3F1ZXVlIG9wdGlvbiBpbiBbY2VsZXJ5XSBoYXMgYmVlbiBtb3ZlZCB0byB0aGUgZGVmYXVsdF9xdWV1ZSBvcHRpb24gaW4gW29wZXJhdG9yc10gLSB0aGUgb2xkIHNldHRpbmcgaGFzIGJlZW4gdXNlZCwgYnV0IHBsZWFzZSB1cGRhdGUgeW91ciBjb25maWcuCi91c3IvbG9jYWwvYWlyZmxvdy8ubG9jYWwvbGliL3B5dGhvbjMuNy9zaXRlLXBhY2thZ2VzL2FpcmZsb3cvY29uZmlndXJhdGlvbi5weTozNTcgRGVwcmVjYXRpb25XYXJuaW5nOiBUaGUgZGVmYXVsdF9xdWV1ZSBvcHRpb24gaW4gW2NlbGVyeV0gaGFzIGJlZW4gbW92ZWQgdG8gdGhlIGRlZmF1bHRfcXVldWUgb3B0aW9uIGluIFtvcGVyYXRvcnNdIC0gdGhlIG9sZCBzZXR0aW5nIGhhcyBiZWVuIHVzZWQsIGJ1dCBwbGVhc2UgdXBkYXRlIHlvdXIgY29uZmlnLgo=" | base64 --decode
/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:357 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:357 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.

score 0 · Accepted Answer · answered Dec 06 '22 at 09:16

This issue ended up being a problem with the MWAA 2.2.2 CLI API where the warning was returned on stderr but no actual error occurred. I ended up adding

while IFS= read -r line; do
    if [[ $line != *"DeprecationWarning"* ]]; then
        ACTUAL_ERROR=1
        break
    fi
done <<< "$ERRORS"

To our shell scripts and

std_err = base64.b64decode(response.json()["stderr"]).decode("utf8").splitlines()
if std_err:
    for line in std_err:
        if "DeprecationWarning" not in line:
            raise ValueError(f"Hit an Airflow error with err message: {std_err}")

to our Python scripts and this solved our core issue.

Amazon Managed Airflow 2.2.2 Fails To Run Tasks From The CLI With DeprecationWarning

Context

DAGs Running In UI

Example MWAA CLI Call

1 Answers1