Handling high number of configuration parameters in python

Question

In my Machine Learning project I have a high number of parameters that are loaded from a configuration file, e.g. a YAML file. I wonder, is there any best practice on how to integrate them in the codebase other than a number of 'setup_by_cfg' functions? I was thinking about classmethods, but then the implementation gets coupled to the parameter file which could be problematic?

# option A
# setup_by_cfg.py

def setup_a(cfg):
    return A(a=cfg.a, b=cfg.b)

def setup_b(cfg):
    ...

# option B
# coupled in class implementation

class A:
    # ...
    @classmethod
    def from_cfg(cls, cfg):
        return cls(a=cfg.a, b=cfg.b)


class B:
    # ...
    @classmethod
    def from_cfg(cls, cfg):
        # ...

Jasha · Answer 1 · 2022-09-29T04:08:49.097

Hydra's instantiate API might be a good fit for you. Using this API, you'd create instances of class A and class B by calling instantiate on configs that have a special _target_ key:

from hydra.utils import instantiate

...

a_instance = instantiate(cfgA)
b_instance = instantiate(cfgB)

Here is a complete example:

from hydra.utils import instantiate
from omegaconf import OmegaConf


class A:
    def __init__(self, x: int, y: int) -> None:
        self.x = x
        self.y = y

yaml_data = """
_target_: __main__.A
x: 123
y: 456
"""

cfgA = OmegaConf.create(yaml_data)

a = instantiate(cfgA)
assert isinstance(a, A)
assert a.x == 123
assert a.y == 456

Handling high number of configuration parameters in python

1 Answers1