How to gather config files in a list with Hydra-fb?

Question

Say I have an abstract class db in my code and classes db1, db1, ... db1 that inherit from db. My project uses hydra and has this structure:

├── my_app.py
├── conf.yaml
└── db
    ├── db1.yaml
    ├── db2.yaml
    └── db3.yaml

I need a list of db so I would like to get a final configuration file like this :

db:
  -
    param1_of_db1: key_1_1
    param2_of_db1: key_1_2
  -
    param1_of_db2: key_2_1
    param2_of_db2: key_2_2
  -
    param1_of_db3: key_3_1
    param2_of_db3: key_3_2

so that db is a list of params of db1, db2, db3. In conf.yaml file, I imagine something like:

defaults:
  - db: [db1, db2, db3]

Is there a way to do something like that ?

score 5 · Accepted Answer · answered Nov 12 '20 at 17:45

What you are asking for is not supported by Hydra.

List composition is not supported, lists are all or nothing during composition.
Config groups are mutually exclusive, there is a feature request to relax that.

You can get close to it though (without being able to override the structure from the command line is something like: config.yaml:

defaults:
  - db/db1
  - db/db2
  - db/db3

This syntax is documented here.

In each db config file you can do something like:

db/db1.yaml:

# @pacakge _group_._name_
host: localhost
port: 3306

Package overrides are documented here.

The resulting config would looke like:

db:       # from the config group of the corresponding config (path)
  db1:    # from the name of the corresponding config
    host: localhost
    port: 3306
  db2:
    ...

btw do you plan to include list composition in next version of Hydra ? — dallonsi, Nov 13 '20 at 09:01
No plans, in 99% of the cases the best thing to do is to replace a list with the second list on merge. — Omry Yadan, Nov 14 '20 at 10:19

score 2 · Answer 2 · answered May 24 '23 at 19:06

2

Actually it seems like it is supported by Hydra now (since Hydra 1.1 ): https://hydra.cc/docs/patterns/select_multiple_configs_from_config_group/

defaults:
  - db:
    - db1
    - db2
    - db3

answered May 24 '23 at 19:06

Gab

21
1

score 1 · Answer 3 · answered Jun 06 '21 at 16:19

You can get pretty close to list merging with the latest hydra/omegaconf

The trick is that dictionaries are composed, so you can compose the config in a dictionary and then use the new oc.dict.values interpolation to get the final list.

So in your case it would be something like:

defaults:
  - dbs/db1
  - dbs/db2
  - dbs/db3

db: ${oc.dict.values:dbs}

Note that I've renamed the "db" package to "dbs". So the final resolved config would look like:

dbs:
  db1:    
    host: localhost
    port: 3306
  db2:
    host: localhost
    port: 3307
  db3:
    host: localhost
    port: 3308

db:
  - host: localhost
    port: 3306
  - host: localhost
    port: 3307
  - host: localhost
    port: 3308

The keys for the "dbs" package aren't used for anything but I find that it actually makes the config clearer to have them.

How to gather config files in a list with Hydra-fb?

3 Answers3

Linked