Referring to Hydra's `conf` directory from a Python sub/sub-sub-directory module

Question

Suppose that we have a Python project with this structure:

hydra_config
├── conf
│   ├── api_key
│   │   ├── non_prod.yaml
│   │   └── prod.yaml
│   └── db
│       ├── mysql.yaml
│       └── postgresql.yaml
├── modules
│   └── module.py
└── my_app.py

Now, in Hydra's config documentation, they state that we need to add a Python decorator on top of a function which we want to give access to the configuration files. However, the docs only showed how to do this to a function in my_app.py which is the main module of the project.

The question is, how would one add the

@hydra.main(config_path="conf")

Python decorator to a function, let's say module_function which is located in modules/module.py? Here's the content of module.py:

import hydra
from omegaconf import DictConfig, OmegaConf

@hydra.main(config_path="conf")
def module_function(cfg: DictConfig):
    print(OmegaConf.to_yaml(cfg))

And then below is the content of the main Python module my_app.py:

from modules.module import module_function

def main():
    module_function()

if __name__ == "__main__":
    main()

When I tried running the main Python module my_app.py with python my_app.py, I instantly got an error saying

Primary config module 'modules.conf' not found.
Check that it's correct and contains an __init__.py file

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

I understand that this means the decorator added to module_function inside module.py couldn't find the conf directory which contains api_key and db config groups.

Does anyone here have any experience with this and know how to fix this error?

Answer 1

First I'll go over some possible solutions, then I'll give an explanation. Here are three things you can do to make your example work:

Fix 1: create conf/__init__.py and change config_path="conf" to config_path="../conf":

├── conf
│   ├── config.yaml
│   └── __init__.py
├── modules
│   └── module.py
└── my_app.py

# modules/module.py
import hydra
from omegaconf import DictConfig, OmegaConf

@hydra.main(config_path="../conf", config_name="config")  # relative path
def module_function(cfg: DictConfig):
    print(OmegaConf.to_yaml(cfg))

Fix 2: move conf to modules/conf and create modules/conf/__init__.py:

├── modules
│   ├── conf
│   │   ├── config.yaml
│   │   └── __init__.py
│   └── module.py
└── my_app.py

Fix 3: define your decorated function in the module that will be called as __main__:

├── conf
│   └── config.yaml
└── my_app.py

# my_app.py
import hydra
from omegaconf import DictConfig, OmegaConf

@hydra.main(config_path="conf", config_name="config")
def module_function(cfg: DictConfig):
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    module_function()

Fix 3 will not be very satisfying to the OP :)

Explanation

In your example, since you've imported the decorated function (rather than defining it in the __main__ module), Hydra is interpreting the argument config_path="conf" as being relative to the parent of the module that defines the decorated function. In your example, the decorated function is defined in modules.module, whose parent is modules, so Hydra is looking for a python package called modules.conf. If found, this package will be searched for yaml files.

Fix 2 above takes the approach of making sure that modules/conf/__init__.py exists, so that modules.conf is in-fact a python package. Meanwhile, Fix 1 takes the approach of using a relative path config_path="../conf". Hydra then looks for a top-level package named conf. Since conf/__init__.py exists, this top-level conf package is found and searched for yaml files. To discover this package, Hydra uses the python import machinery. This means Hydra has the ability to discover yaml files located in python packages that are elsewhere on your $PYTHONPATH or that are installed via a package manager such as pip.

In the case of Fix 3, we have module_function.__module__ == "__main__", which means that Hydra cannot rely on python's import machinery to discover where module_function was defined. As a fallback, Hydra tries to find a directory conf (rather than a package conf); this is why an __init__.py file is not required for Fix 3. Here are the key lines of code implementing this behavior. Note that conf is still treated as a relative path, relative to the directory containing the file that defined the decorated function (rather than to the parent of the module that defined it). Here is the function implementating this relative path computation (for both the module case and the directory case).