MWAA load custom file info to DAG

Question

I'm trying to use a file in a DAG. The codes I want to use is basically this:

conf_device_info = OmegaConf.load(f"./config/{dag_name}/config_dtype.json")

and my bucket is currently like this:

my-bucket
--/ dags
    -- /config
           --/{dag_name}
                      --/config_dtype.json
    -- dag_with_the_code.py
    -- /utils
           --s3_manager.py

When I import s3_manager with "import utils.s3_manager" , it goes fine. When I try to run the code the OmegaConf code, it says

  FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/airflow/config/{dag_name}/config_dtype.json'

What should I do to do what I'm trying to acheive? Why is import working and referencing file with absolute path not working..? Thanks in advance.

score 0 · Answer 1 · answered Mar 09 '23 at 09:17

In the airflow documentation there is a section about relative imports

When I'm importing config files, I usually use something like

from pathlib import Path
CONFIG_PATH = Path(__file__).parent / "include" / "config.yaml"

with open(CONFIG_PATH, encoding="UTF-8") as config_file:
    config = yaml.safe_load(config_file)

Or you can use airflow environment variables for the location of the DagBag

MWAA load custom file info to DAG

1 Answers1