0

I want to dynamically alter pipelines based on the configuration provided. Is there possibility to pass configuration based on the environment to the register_pipelines() or to the create_pipeline() functions?

I've read the documentation about configuration (https://kedro.readthedocs.io/en/stable/kedro_project_setup/configuration.html) but there is not specified how can one get the necessary information to use ConfigLoader it (mainly environment) that was used in kedro run --env= command.

E.g. Similarly when creating nodes, is it possible to create pipelines with create_pipeline("params:model_params")?

WestFlame
  • 435
  • 1
  • 7
  • 16

2 Answers2

2

I've finally found that there is (imho not mentioned in the documentation) hook after_context_created (https://github.com/kedro-org/kedro/blob/main/kedro/framework/hooks/specs.py#L289), which I can use to fit my needs.

WestFlame
  • 435
  • 1
  • 7
  • 16
  • This is indeed the right way to do it! See https://github.com/kedro-org/kedro/discussions/1436#discussioncomment-2789761 and following answer there for a very similar case. `after_context_created` is very new and hasn't quite made it into the documentation yet but will be there soon :) – Antony Milne Jun 23 '22 at 08:51
  • 2
    @WestFlame could you please share your solution, or add the snippet for ```after_context_created``` hooks here? – Saeid Hedayati Jun 30 '22 at 12:44
0

To take advantage of configuration environments simply create a new folder under conf/ e.g. conf/prod and then running kedro run --env prod will run things with the configuration under that folder.

The environment name should also be available in many of the hooks you can extend kedro with, although the node itself shouldn't be aware of which environment is currently configured.

datajoely
  • 1,466
  • 10
  • 13
  • Thanks, but what I wanted to know is how can I access loaded configuration while creating the pipelines? The documentation only shows examples for the nodes. – WestFlame Jun 21 '22 at 14:14