MWAA (aws managed airflow) failed to pause DAG when using CLI

Question

I am trying to pause a DAG as described here but getting 'Dag id testDag not found' even though the DAG exists.

The error message says that my 3rd party module 'not found' even though it works when I trigger the DAG.

I saw this on the doc :

Note Any command that parses a DAG (such as list_dags, backfill) will fail if the DAG uses plugins that depend on packages that are installed through requirements.txt.

Dag code (testDag):

from datetime import timedelta
# my_client is the 3rd party library
from client import my_client
from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.operators.python_operator import PythonOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': days_ago(2),
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_retry': False,
}
dag = DAG(
    'testDag',
    default_args=default_args,
    description='A simple tutorial DAG',
    schedule_interval=None,
    catchup=False
)

def print_me_func(**context):
    print("hiiiii")

printMe = PythonOperator(
    task_id='printMe',
    dag = dag,
    provide_context=True,
    python_callable=print_me_func,
)

Airflow API :

import boto3
import json
import requests
import base64

mwaa_env_name = 'YOUR_ENVIRONMENT_NAME'
dag_name = 'testDag'

client = boto3.client('mwaa')

mwaa_cli_token = client.create_cli_token(
    Name=mwaa_env_name
)

mwaa_auth_token = 'Bearer ' + mwaa_cli_token['CliToken']
mwaa_webserver_hostname = 'https://{0}/aws_mwaa/cli'.format(mwaa_cli_token['WebServerHostname'])
raw_data = "pause {0}".format(dag_name)

mwaa_response = requests.post(
    mwaa_webserver_hostname,
    headers={
        'Authorization': mwaa_auth_token,
        'Content-Type': 'text/plain'
    },
    data=raw_data
)

mwaa_std_err_message = base64.b64decode(mwaa_response.json()['stderr']).decode('utf8')
mwaa_std_out_message = base64.b64decode(mwaa_response.json()['stdout']).decode('utf8')

print(mwaa_response.status_code)
print(mwaa_std_err_message)
print(mwaa_std_out_message)

200
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 76, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 421, in pause
    set_is_paused(True, args)
  File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 431, in set_is_paused
    is_paused=is_paused,
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dag.py", line 1887, in set_is_paused
    raise DagNotFound("Dag id {} not found".format(self.dag_id))
airflow.exceptions.DagNotFound: Dag id testDag not found

[2021-03-23 20:37:45,790] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/testDag.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/airflow/dags/testDag.py", line 2, in <module>
    from client import my_client
ModuleNotFoundError: No module named 'client'
[2021-03-23 20:37:45,802] {{dagbag.py:259}} ERROR - Failed to import: /usr/local/airflow/dags/testDag.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 256, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/usr/lib64/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/airflow/dags/testDag.py", line 2, in <module>
    from client import my_client
ModuleNotFoundError: No module named 'client'

Any workarounds that I can use ? Thanks

Add your DAG code and the specific error traceback – Elad Kalif Mar 23 '21 at 19:15 — Elad Kalif, Mar 23 '21 at 19:15

score 0 · Answer 1 · answered Jun 28 '21 at 20:23

0

Seems like MWAA 1.10.12 has a serialization issue, the same code works when migrating from MWAA 1.10.12 to MWAA 2.0.2.

answered Jun 28 '21 at 20:23

user3099614

19
1
3

score -1 · Answer 2 · answered Oct 25 '22 at 20:40

-1

I am using 2.2.2 and still have the same issue, looks like the plugins are not getting installed to web server where the MWAA cli cmds are getting executed.

answered Oct 25 '22 at 20:40

SaiHarsha Tallapalli

1

This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://stackoverflow.com/questions/ask). To get notified when this question gets new answers, you can [follow this question](https://meta.stackexchange.com/q/345661). Once you have enough [reputation](https://stackoverflow.com/help/whats-reputation), you can also [add a bounty](https://stackoverflow.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/late-answers/33029605) – configbug Oct 31 '22 at 17:39

MWAA (aws managed airflow) failed to pause DAG when using CLI

2 Answers2