No module named ____ error in AWS MWAA even with requirements.txt

Question

I've uploaded my requirementst.txt to a dag and referenced it when creating my Airflow Environment. Here is what my requirements.txt looks like:

apache-airflow==1.10.12
oauth2client==4.1.3
google-api-python-client==2.2.0
Flask-AppBuilder==2.3.3
boto3==1.17.59

However, I keep getting a "No module named oauthclient" error in my airflow environment. How do I check if oauthclient is actually installed or if Airflow is not reading my requirements.txt file correctly?

Thanks in advance.

score 4 · Answer 1 · edited Jun 15 '21 at 11:36

4

You don't need to recreate MWAA. MWAA uses versioned S3 bucket. Your requirements.txt is also versioned and stick with the old version if you don't manually make it, point to the latest version.

Just go to MWAA page, click "edit", and choose the latest version of your requirements.txt. Then it will update.

edited Jun 15 '21 at 11:36

bad_coder

11,289
20
44
72

answered Jun 11 '21 at 08:42

marsan mars

41
2

score 1 · Answer 2 · answered Dec 02 '21 at 13:59

Updating requirements in the MWAA is not straightforward:

Once the requirements text is changed , you have to rebuild the env again
Before you do this , please double check to make sure that you have selected the right version of the requirements .txt file in the environment creation page
Most of the time the version of .txt file is the older one, you have to manually select the newest one!
Simply editing and updating the existing environment will install the older version of the requirements.txt file!

score 0 · Accepted Answer · edited May 16 '22 at 01:16

0

I solved this by recreating my environment. It looks like AWS MWAA only runs your requirements.txt once and so any updates you make after you have made your environment will not be applied.

edited May 16 '22 at 01:16

MattG

5,589
5
36
52

answered May 05 '21 at 16:30

semiflex

1,176
3
25
44

it seems like this is true but it cannot be – JustEngland Aug 26 '21 at 15:29
I just spoke with AWS on this issue, looks like it just takes some time to see the changes. You should be able to see the requirements file running in the scheduler logs. – JustEngland Aug 26 '21 at 16:19

score 0 · Answer 4 · answered Nov 23 '21 at 21:05

0

mwaa should read the requirement.txt again, if you edit the mwaa configuration through the console and press save or from cli, run: aws mwaa update-environment --name <environment name>

answered Nov 23 '21 at 21:05

Tom Sarpi

29
3

No module named ____ error in AWS MWAA even with requirements.txt

4 Answers4