0

Background

I currently have a script which does some API data gathering. The script is deployed to two environments, test and user. Each environment has different settings which I have created the respective configuration files. I am currently migrating my project to use the hydra-core package.

Objective

If possible I would like configure Hydra as such. Load all the configuration files from either the test or user. The default should be the test environment unless specified via CLI to load the user environment.

Current Configuration Structure

Project Root
|
├── config
│   ├── config.yaml
│   ├── test
│   │   ├── config1.yaml
│   │   ├── config2.yaml
│   │   ├── config3.yaml
│   │   └── config4.yaml
│   └── user
│       ├── config1.yaml
│       ├── config2.yaml
│       ├── config3.yaml
│       └── config4.yaml

Current Default Configuration - config.yaml

defaults:
  - test: [config1.yaml, config2.yaml, config3.yaml, config4.yaml]

How would I override the above default via the CLI?

Anthon
  • 69,918
  • 32
  • 186
  • 246
Raymond C.
  • 572
  • 4
  • 24

1 Answers1

1

You can introduce a config group, e.g. env that would contain configs with the default lists you want to use for each environment.

├── config
│   ├── config.yaml
│   ├── env
│   │   ├── user.yaml
│   │   └── test.yaml
│   ├── test
│   │   └── user.yaml
│   └── user
│       └── config1.yaml

env/user.yaml:

defaults:
 - /user/config1
 - /user/config2

env/test.yaml:

defaults:
 - /test/config1
 - /test/config2

Primary config.yaml:

defaults:
 - env: test # Overridable

With the above, you should be able to override the environment like:

$ python foo.py env=user

This is similar to the configure experiment pattern. This is relying on recursive default lists, new in Hydra 1.1. You can learn more about them here

Omry Yadan
  • 31,280
  • 18
  • 64
  • 87
  • Based on my current implementation, I already have split them into the different `env` which is like a config group. How would I specify via CLI which `env` to use? Thanks! – Raymond C. May 12 '22 at 10:01
  • I updated the answer and added another link. Please read both links before implementing. – Omry Yadan May 13 '22 at 04:01
  • Thanks for the edit, I have tested it and works just as I wanted. – Raymond C. May 13 '22 at 06:18