How to use a config group multiple times, while overriding each instance

Question

Here is my current config structure

hydra/
  pipeline/
    common/
      feature.yaml
  stage/
    train.yaml

with the following files:

train.yaml

# @package _global_
defaults:
  - _self_
  - ../pipeline/common@train: feature
  - ../pipeline/common@val: feature

train:
  conf:
    split: train

val:
  conf:
    split: val

pipeline:
  - ${oc.dict.values: train.steps}
  - ${oc.dict.values: val.steps}

feature.yaml

conf:
  split: train

steps:
  tabular:
    name: "${conf.split}-tabular
    class: FeatureGeneration
    dataset:
      datasources: [ "${conf.split}_split" ]

What I've accomplished:

• I've been able to figure out how to use the config group multiple times utilizing the defaults in train.yaml.

What I'm stuck on:

1. I'm getting an error: InterpolationKeyError 'conf.split' not found I do realize that imports are absolute. If I put @package common.feature at the beginning of feature.yaml I can import conf.split via common.feature.conf.split, but is there not a cleaner way? I tried relative imports but got the same error.

2. I can't seem to override conf.split from train.yaml. You can see where I set train.conf.split and val.conf.split but these do not get propagated. What I need to be able to do is have each instance of the config group utilize a different conf.split value. This is the biggest issue I'm facing.

What I've referenced so far:

The following resources have gotten me to where I am so far, but am still having trouble with what's listed above.

• Hydra : how to assign config files from same group to two different fields
• https://hydra.cc/docs/advanced/overriding_packages/
• https://hydra.cc/docs/patterns/extending_configs/

Answer 1

1. Interpolation is not import and it's evaluated at when you access the config node. At that point your config is already composed so it should be straight forward to use either absolute interpolation (the default) or relative based on the structure of your final config.

2. Hard to be 100% sure, but I suspect this problem is because your defaults list has _self_ at the beginning. This means that the content of the config with containing the defaults list is overridden by what comes after in the defaults list. Try to move _self_ to the end:

# @package _global_
defaults:
  - ../pipeline/common@train: feature
  - ../pipeline/common@val: feature
  - _self_
#...