0

I'd like to use specialized configuration as per Hydra documentation in Common Patterns -> Specializing Configuration. The difference is that my specialized configuration is in a file, not just one variable. In the example below I want to choose transform based on the model and the dataset. The configs for different transforms are in files. This would work if I specified all the transform configuration in dataset_model/cifar10_alexnet.yaml file, but that would defeat the purpose because I can't reuse the transform config in this case. Alsewhere in Hydra if you specify the name of the file it would automatically pick up the config in that file, but it does not seem to work in the specialized configuration.

I've modified the example in documentation as follows:

config.yaml:

defaults:
  - dataset: cifar10
  - model: alexnet
  - transform: crop
  - dataset_model: ${defaults.0.dataset}_${defaults.1.model}
    optional: true

Added directory called transform and two files inside that directory:

crop.yaml:

# @package _group_
type: crop
test1: 7

resize.yaml:

# @package _group_
type: resize
test1: 50

changed file dataset_model/cifar10_alexnet.yaml:

# @package _global_
model:
  num_layers: 5
transform: resize

Everything else is exactly as per the documentation. When I run this I get an exception:

Traceback (most recent call last):
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/hydra/_internal/config_loader_impl.py", line 720, in _merge_config
    ret = OmegaConf.merge(cfg, loaded_cfg)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 321, in merge
    target.merge_with(*others[1:])
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 331, in merge_with
    self._format_and_raise(key=None, value=None, cause=e)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/base.py", line 101, in _format_and_raise
    type_override=type_override,
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
    _raise(ex, cause)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 329, in merge_with
    self._merge_with(*others)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 347, in _merge_with
    BaseContainer._map_merge(self, other)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/basecontainer.py", line 296, in _map_merge
    dest.__setitem__(key, src_value)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 262, in __setitem__
    self._format_and_raise(key=key, value=value, cause=e)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/base.py", line 101, in _format_and_raise
    type_override=type_override,
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 694, in format_and_raise
    _raise(ex, cause)
  File "/home/natalia/.pyenv/versions/3.7.9/lib/python3.7/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ValidationError: 
    full_key: transform
    reference_type=Optional[Dict[Union[str, Enum], Any]]
    object_type=dict

So, the question is - is this functionality supported and if it is, what am i doing wrong?

Natalia Levine
  • 3
  • 3

1 Answers1

0

Your config is trying to merge the string "resize" into a dictionary like:

transform:
  type: crop
  test1: 7

This is not something you can do.

You are not explaining what you are trying to do very well, but my guess is that you want to compose a different transform based on selected dataset.

Hydra 1.1 will add support for recursive defaults list which will probably allow you to do what you want.

This is the doc for the new defaults list. You can install this version as a pre-release (see primary project readme).

Omry Yadan
  • 31,280
  • 18
  • 64
  • 87
  • Added a bit more explanation to the question. What I'm trying to do is very simple - pick the transform based on the model and dataset. The different transform configs are in separate files. Everywhere else you support specifying the file name, but it looks like the specialized config does not support it. I red through your documentation for the defaults list and I don't see how it is related to this use case. Could you please explain how you would use the recursive defaults list for this? – Natalia Levine Feb 22 '21 at 09:54
  • Can you show me one more example where I support specifying a file name? config names can only be specified in the Defaults List. – Omry Yadan Feb 23 '21 at 06:04
  • I think you some misconception about how Hydra works. I suggest you go over the basic tutorial to understand things better before you try to use features like interpolation in the defaults list ("Specialized configs") – Omry Yadan Feb 23 '21 at 06:06
  • I took a look and what you are after is not supported in a satisfactory way even in Hydra 1.1. You have a misconception that makes you think specifying config names in the config is "supported everywhere" where in fact it's not supported anywhere. It is possible to achieve something reasonable in 1.1 but it's not very elegant. – Omry Yadan Feb 23 '21 at 07:07