Importing `YAML` files inside `experiment.yaml` files

Question

Context: using hydra to run experiments with several settings in yaml files.

Assume workd_dir is defined in the inference_root.yaml file, e.g.:
work_dir: ${hydra:runtime.cwd}

Further assume the following experiment.yaml structure, where we would like to load settings defined in other YAML files and not in the inference_root.yaml file:

experiment_specific_settings:
  option_1: ${work_dir}/setting_v1.yaml
  option_2: ${work_dir}/setting_v2.yaml

This will fail to parse the referenced YAML files. However, adding the following:

def include_constructor(loader, node):
    # Get the included file path from the YAML node
    filename = loader.construct_scalar(node)

    # Load the included file
    included_yaml = yaml.load(open(filename), Loader=yaml.SafeLoader)

    return included_yaml


# Add the constructor to the SafeConstructor class
yaml.constructor.SafeConstructor.add_constructor('!include', include_constructor)

Combined with placing !include in front of option_1 and option_2 will.

Unfortunately, the yaml evaluation happens before the relative paths are evaluated by hydra, e.g., instead of yaml evaluating settings_folder/setting_v1.yaml, it will see ${work_dir}/setting_v1.yaml`, which it won't find.

How do I either:

1. read in nested yaml file references directly using hydra, or;
2. ensure the relative run-time paths are evaluated before the yaml is parsed?

---

EDIT per commenter request:

Assume the following directory structure:

configs/
├── inference_root.yaml
└── experiments/
    ├── my_experiment.yaml
    └── ...
└── tires/
    ├── ....yaml
    └── ....yaml
└── engines/
    ├── ....yaml
    └── ....yaml

Experiments let two cars race against each other, car_1 and car_2. Each car consists of several parts, e.g., tires, engines, doors, each with their own specifications, e.g., tires_1.yaml, tires_2.yaml, ... etc.

To ensure repeatable experiments, ideally we would like to structure experimental setup as follows:

inference_root.yaml

defaults:
  experiments: my_experiment

my_experiment.yaml

car_1:
  tires: tires_1
  engine: engine_3
  doors: door_81

car_2:
  tires: ...
  engine: ...
  doors: ...

Problems:

1. Currently, I am not able to retrieve a cfg in my app.py, that loads all the nested .yaml files referenced in the my_experiment.yaml.
2. Assume that the nested YAML files are not in the configs, but in a different directory who's path is determined at run-time using the following lines in inference_root.yaml:

# path to work directory
work_dir: ${hydra:runtime.cwd}
# path to data directory
data_dir: ${work_dir}/data

Is there still a way to auto-load these YAML files?

Any guidance would be appreciated!

Answer 1

To compose the following in my_experiment.yaml:

car_1:
  tires: tires_1
  engine: engine_3
  doors: door_81

car_2:
  tires: ...
  engine: ...
  doors: ...

Try something like this:

# @pacakge _global_
defaults:
- /tires@car_1: tires_1
- /engine@car_1: engine_3
- /doors@car_1: door_81
- /tires@car_2: tires_1
- /engine@car_2: engine_1
- /doors@car_2: door_1

The following pages can help you understand what is going on:

1. Configuring experiments
2. Packages

Note that I did not try it. If you get something unexpected comment and I will try to adjust.