How to use hydra compose API to read yaml from a nested directory?

Question

I am trying to print the src.yaml from a nested directory below

Main_dir:
 conf:
  project:
    test_project1:
      env:
        source:
            src.yaml

src.yaml

table: TEST_TABLE
DB:Test_DB

I tried the following hydra API

with initialize(version_base=None, config_path="../../conf"):
 cfg = compose(config_name="config", overrides = ["project=test_project1","+source=src"])
 print(cfg)

but I am getting the following error:

hydra.errors.ConfigCompositionException: Could not override 'project'. No match in the default list

How can I get rid to of this error?

your `src.yaml` is invalid. Either you have to indent the continuation line of the unquoted scalar "TEST_TABLE DB:Test_DB" or you need to make it two key-value pairs in the root level mapping by inserting a space after the colon to make it a value indicator — Anthon, May 26 '23 at 19:10
Hi @Anthon.. Thanks for your replay.. In actual scenario the application reads yaml directly and there is no error as you mentioned.. This is the dummy yaml.. The issue now i am facing here is traversing through the directory to read the yaml and i am not sure how to override that — Hari, May 27 '23 at 10:53

Omry Yadan · Accepted Answer · 2023-05-30T13:15:48.563

project=test_project1 means that there is a CONFIG named test_project1. Not a directory.

The Compose API gives you the same config composition capabilities @hydra.main() gives you. What you are trying to do is not supported in that form.

Something like this is more likely to work:

with initialize(version_base=None, config_path="../../conf"):
 cfg = compose(config_name="config", overrides = [
  "project/test_project1/env/source=src"
 ])
 print(cfg)

You are providing the conf config directory as your config path. This means all the sub directories are considered config groups.

I suggest you go back and read the Hydra basic tutorial. If you want to get deeper, I recommend The Defaults List.

How to use hydra compose API to read yaml from a nested directory?

1 Answers1