All DAGs broken after MWAA update from 2.0.2 to 2.2.2

Question

I am getting the following errors in AWS MWAA UI after I updated from 2.0.2 to 2.2.2 I have exhaustively searched for more details on these errors to no avail

from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator
ModuleNotFoundError: No module named 'airflow.providers.slack'

Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/.___init__.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/operators/._aws_glue_crawler_operator.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/operators/._aws_glue_job_operator.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/operators/.___init__.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/hooks/.___init__.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/sensors/._aws_glue_job_sensor.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/sensors/.___init__.py] source code string cannot contain null bytes

The CloudWatch logs from the WebServer are no helpful either:

ValueError: source code string cannot contain null bytes

PFA

My plugins zip file while I was using 2.0.2 was of the following structure and it worked just fine. I believe this is not working for v2.2.2:

s3://{my-bucket-name}
└── DAGS
    ├── dags 
    │   └── dag1.py  
    └── plugins
        ├── __init__.py
        └── operators
        |   ├── __init__.py
        |   └── aws_glue_operator.py
        |___ sensors
             |__aws_glue_job_sensor.py

Here is my requirements.txt contents

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt"
apache-airflow-providers-postgres[amazon]
apache-airflow-providers-cncf-kubernetes
psycopg2-binary==2.8.6
fsspec
s3fs
pandas
sagemaker==v1.72
dag-factory==0.7.2
requests
boto3
botocore
apache-airflow-providers-amazon
PyOpenSSL
oauth2client
gspread

Any help is appreciated

Have you installed the `apache-airflow-providers-slack` 4.1.0 or different version (What version do you tried to install)? How are you specifying your dependencies? — Emma, Jul 25 '22 at 15:12
The error shows that you are using slack plugin but you are missing the `apache-airflow-providers-slack==4.1.0` in your `requirements.txt`. That can solve the last DAG's issue. Other issues seems from the aws glue related files. I will update if I find the fix for this but please first try to include slack plugin and see if error reduces. — Emma, Jul 26 '22 at 13:14
I am not sure but do you happen to have dot files (**.**_xxx.py) in S3? I see the error points to hidden file(`._aws_glue_job_sensor.py`). If so, remove these files. — Emma, Jul 26 '22 at 13:24

score 0 · Accepted Answer · answered Jul 26 '22 at 16:14

0

With MWAA 2.2.2 the plugins zip file and the requirements.txt need to be downloaded from here Any version conflicts in requirements.txt and Airflow should be cross-checked from here

With these two files I was able to fix the broken DAGs in 2.2.2

answered Jul 26 '22 at 16:14

ab_padfoot

63
1
10

score 0 · Answer 2 · answered Oct 19 '22 at 20:27

Using a public VPC (but tested first on the local runner container) I was able to get Slack to work by just adding the following to the requirements.txt file:

apache-airflow-providers-slack==5.0.0

This version worked without issue.

You can enable logging (in the Monitoring section) and then update and check the logs it produces. You should see a successful installed message in the CloudWatch logs (requirements_install_ Log stream).

All DAGs broken after MWAA update from 2.0.2 to 2.2.2

2 Answers2