How to reference other .yaml files of hydra conf folder?

Question

My conf directory looks like:

conf
- hydra
 - run
   - dir
    - job_timestamp.yaml
main.yaml

in main.yaml I am trying to overwrite the hydra output directory with a custom structure as defined in job_timestamp.yaml:

dir: ./outputs/${status}/${now:%Y-%m-%d}/${now:%H-%M-%S}

However, writing:

hydra:
  run:
    dir: job_timestamp

in main.yaml doesn't reference the hydra/run/dir/job_timestamp.yaml file I have - it actually just makes the output folder name "job_timestamp"

I was wondering how I would reference my job_timestamp.yaml file in main.yaml to overwrite hydra's output directory config.

EDIT: I originally had hydra/run/dir: job_timestamp under defaults of main.yaml, however, it would return with hydra/run/dir not found in defaults list. I am planning to have different versions of main.yaml (e.g. main2.yaml, evaluate.yaml etc.) which all would need to override the output directory with the same format, is there a way to do this so that it is DRY?

Answer 1

You seem to be confusing basic concepts in Hydra like config values and config groups. You do not "reference" a config file from a config value, only from the defaults list.

Go back to the basic tutorial and make sure you understand config groups. Hydra specific config groups are defined inside the Hydra package (here). You may want to override Hydra configs via config groups - but I only suggest you attempt that after you have a better understanding of config groups. In the mean time, you can do it by simply overriding it directly in your main.yaml file:

main.yaml:

hydra:
  run:
    dir: outputs/${status}/${now:%Y-%m-%d}/${now:%H-%M-%S}