How to access environment name in kedro pipeline

Question

Is there any way to access the kedro pipeline environment name? Actually below is my problem.

I am loading the config paths as below

conf_paths = ["conf/base", "conf/local"]  
conf_loader = ConfigLoader(conf_paths)
parameters = conf_loader.get("parameters*", "parameters*/**")
catalog = conf_loader.get("catalog*")

But I have few environments like "conf/server" , "conf/test" etc, So if I have env name available I can add it to conf_paths as "conf/<env_name>" so that kedro will read the files from the respective env folder. But now if the env path is not added to conf_paths, the files are not being read by kedro even if i specify the env name while I run kedro like kedro run --env=server I searched for all the docs but was not able to find any solution.

EDIT: Elaborating more on the problem. I am using the above-given parameters and catalog dicts in the nodes. I only have keys that are common for all runs in conf/base/parameters.yml and the environment specific keys in conf/server/parameters.yml but when i do kedro run --env=server I am getting keyerror which means the keys in conf/server/parameters.yml is not available in the parameters dict. If I add conf/server to config_paths kedro is running well without keyerror.

You can always set env variable externally and read using `os.environ['ENV_VAR_NAME']`. But it shouldn't be required. Just passing `kedro run --env=server` should work. Can you add more details why you think it is not being read ? — Rahul Kumar, Dec 14 '21 at 22:24
If you're running with `kedro run --env=server`, why are you manually constructing `conf_paths`/the `ConfigLoader`/etc.? I don't know what version you're using, but it gets handled by the Kedro framework in code like https://github.com/quantumblacklabs/kedro/blob/0.17.6/kedro/framework/context/context.py#L419-L422. Here, you can see that it takes the configured environment (without getting into how it propagates down to here from where you initially passed it in the CI, how it went to the Kedro `Session` constructor, etc.). — deepyaman, Dec 14 '21 at 23:54
@deepyaman How can I access parameters in nodes.py without specifically passing the parameters into each node? — DataEnthusiast, Dec 15 '21 at 13:34
When you used `kedro run --env=server`. You didn't define `conf_loader = ConfigLoader(conf_paths)` right ? If yes then it has become hardcoded to only base and local env. — Rahul Kumar, Dec 15 '21 at 18:10
Let it be as it like below, unless you are doing something specific. ``` def register_config_loader( self, conf_paths: Iterable[str], env: str, extra_params: Dict[str, Any] ) -> ConfigLoader: return ConfigLoader(conf_paths) ``` — Rahul Kumar, Dec 15 '21 at 18:12
@Rahulkumar I really don't get your idea. where should I define the function? If possible can you please give an elaborate answer? actually i want to use parameters in the nodes.py — DataEnthusiast, Dec 15 '21 at 19:31
@babu Added the details in the answer. Let me know if it helps. — Rahul Kumar, Dec 15 '21 at 20:28

score 0 · Accepted Answer · answered Dec 15 '21 at 20:27

0

You don't need to define config paths, config loader etc unless you are trying to override something.

If you are using kedro 0.17.x, the hooks.py will look something like this.

Kedro will pass, base, local and the env you specified during runtime in conf_paths into ConfigLoader.

class ProjectHooks:
    @hook_impl
    def register_config_loader(
        self, conf_paths: Iterable[str], env: str, extra_params: Dict[str, Any]
    ) -> ConfigLoader:
        return ConfigLoader(conf_paths)

    @hook_impl
    def register_catalog(
        self,
        catalog: Optional[Dict[str, Dict[str, Any]]],
        credentials: Dict[str, Dict[str, Any]],
        load_versions: Dict[str, str],
        save_version: str,
        journal: Journal,
    ) -> DataCatalog:
        return DataCatalog.from_config(
            catalog, credentials, load_versions, save_version, journal
        )

In question, I can see you have defined conf_paths and conf_loader and the env path is not present. So kedro will ignore the env passed during runtime.

answered Dec 15 '21 at 20:27

Rahul Kumar

2,184
3
24
46

I have the same code in hooks.py. Actually i want to use parameters in nodes.py. How to use that. I tried the below code and its not working ``` context = KedroContext("project_name", os.getcwd()) catalog = context.catalog parameters = catalog.load('parameters')''' when i run ```kedro run --env=server ''' still kedro is loading params from base/local. I am not sure what iam doing wrong. – DataEnthusiast Dec 15 '21 at 21:56
1

You can try this `from kedro.framework.session import get_current_session` `session = get_current_session()` `context = session.load_context()` `context.params` to get the parameters. – Rahul Kumar Dec 15 '21 at 22:11
`catalog.load('parameters')` it means you are loading a catalog called `parameters`. – Rahul Kumar Dec 15 '21 at 22:11
BTW, I didn't understand the need of loading parameters separately in node when you can directly pass as node input. – Rahul Kumar Dec 15 '21 at 22:13
This is working. But I am expecting that the --params i pass from the command line should be added to the existing env parameters. – DataEnthusiast Dec 15 '21 at 22:25
I will make my changes to pass the params directly to the node. The previous developer hard-coded the path as he doesn't have env. Now I need to change all the nodes. – DataEnthusiast Dec 15 '21 at 22:28
The parameter passed using `--params` will work and will be there at runtime. Can you share exactly how are you passing the params in `--params` ? It should be kv pairs like this `--params "key1:value with spaces,key2:value"` – Rahul Kumar Dec 15 '21 at 22:32
I am passing the `--params` as kv pairs as you mentioned. thanks for your help – DataEnthusiast Dec 16 '21 at 14:02

How to access environment name in kedro pipeline

1 Answers1