1

Question

I am currently having trouble with getting hydra to work when not using @hydra.main in the main.py script (refer to error section). I have tried the compose API, but that prevents me from using most of the functionality that hydra provides. I am unsure what I am trying to achieve is possible.

Current Configuration Structure

Project Root
|
├── config
│   ├── default.yaml
│   ├── configuration1.yaml
│   └── configuration2.yaml
└── src
     ├── utils
     │   └── initialise.py
     └── main.py

Code

initialise.py

import hydra
import omegaconf

CONFIG = None

@hydra.main(version_base=None, config_path='../../config', config_name='default')
def init_configs(cfg: omegaconf.DictConfig) -> None:

    global CONFIG
    CONFIG = cfg

    # Other initialisation code here.   

init_configs()

main.py

from utils import initialise

print(initialise.CONFIG)

Error

Primary config module 'config' not found.
Check that it's correct and contains an __init__.py file
Raymond C.
  • 572
  • 4
  • 24

1 Answers1

2

This is an attempt to implement many anti-patterns.

@hydra.main() is designed to be your entry point. Using it a utility function is wrong.

  1. Hydra config composition is very expressive. Allow Hydra to fully initialize your config. Do not "initialize" it further. This is preventing overriding things from the command line.
  2. Do not pass your config via a global. Pass individual config nodes to sub modules/functions. If you need to access the same config variable from many functional modules use interpolation (key: ${foo.bar}).
  3. Attempting to reuse the entry point is a cause of many issues. You didn't write it explicitly, but your code smells a bit like an attempt to reuse the entry point. Don't be afraid to have as many @hydra.main() entry points as you have runnable modules

If you insist on having a config initialization utility, you need to let go of @hydra.main() and use the compose API. You can also combine the two but my recommendation to just let hydra.main() compose your config.

Omry Yadan
  • 31,280
  • 18
  • 64
  • 87
  • Understood point 1. For point 2, do you mean pass the configuration via function parameters? For point 3, I still face the same problem when calling `@hydra.main` from sub modules, it is unable to find the `config` like the error I described. Thanks a lot! – Raymond C. Sep 22 '22 at 13:26
  • When I call `@hydra.main` from my `main.py` and `util/submodule.py` I get this error: ValueError: GlobalHydra is already initialized, call GlobalHydra.instance().clear() if you want to re-initialize – Raymond C. Sep 22 '22 at 13:34
  • Pass the config or a part of it to your function, e.g foo(cfg.bar). About calling hydra.main(): You should call hydra.main only from your main script (otherwise it will not work as you are seeing). – Omry Yadan Sep 22 '22 at 17:54
  • Understood. One last question what is the difference between using compose API and Omegaconf directly? Also does compose API support CLI arguments overriding? – Raymond C. Sep 23 '22 at 00:39
  • 1
    The Compose API is composing a config with the full power of Hydra. Hydra is using OmegaConf as the primitive config API but adds many features like the Defaults List, config discovery and more. The Compose API supports overriding through the callsite (`cfg = compose(config_name="config", overrides=["db=mysql", "db.user=me"])`), if you want to override from the command line it's your responsibility to pass sys.argv in. – Omry Yadan Sep 23 '22 at 13:22