Facing kedro-airflow validation error from pydantic when running in docker

Question

I am new to kedro and airflow. I am trying to deploy a kedro pipeline in airflow by using docker. But while executing my DAG I get this error:

2022-01-27 16:17:19,659 - airflow.task - ERROR - Task failed with exception
Traceback (most recent call last):
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1286, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1316, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/airflow/dags/adag.py", line 39, in execute
    session.run(self.pipeline_name, node_names=[self.node_name])
  File "/home/astro/.local/lib/python3.7/site-packages/kedro/framework/session/session.py", line 410, in run
    run_params=record_data, pipeline=filtered_pipeline, catalog=catalog
  File "/usr/local/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.7/site-packages/kedro_mlflow/framework/hooks/pipeline_hook.py", line 108, in before_pipeline_run
    self._is_mlflow_enabled = _assert_mlflow_enabled(run_params["pipeline_name"])
  File "/usr/local/lib/python3.7/site-packages/kedro_mlflow/framework/hooks/utils.py", line 8, in _assert_mlflow_enabled
    mlflow_config = get_mlflow_config()
  File "/usr/local/lib/python3.7/site-packages/kedro_mlflow/config/kedro_mlflow_config.py", line 228, in get_mlflow_config
    mlflow_config = KedroMlflowConfig.parse_obj(conf_mlflow_yml)
  File "pydantic/main.py", line 511, in pydantic.main.BaseModel.parse_obj
  File "/usr/local/lib/python3.7/site-packages/kedro_mlflow/config/kedro_mlflow_config.py", line 106, in __init__
    super().__init__(**kwargs)
  File "pydantic/main.py", line 331, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for KedroMlflowConfig
hooks
extra fields not permitted (type=value_error.extra)

Please help me in understanding and resolving this issue.

Answer 1

Disclaimer: I am the author of the kedro-mlflow plugin, but I am not a member of the kedro core team and I don't know the internals of the kedro-airflow plugin

Your error is raised by the kedro-mlflow plugin, and it indicates that your mlflow.yml config file has an extra key which is not supported.

The simplest way to make it work again is to generate a new config file with the correct keys with the following command:

kedro mlflow init --force

The other way is to check manually the file to find the extra key you have. I suspect that you generated the file with kedro-mlflow==0.7.x when you created your project, and you try to run the project in a virtual environment with kedro-mlflow==0.8.0 which has been realeased a few weeks ago and which has slightly different keys. In general, I only introduce breaking changes between minor releases (and once I've reached 1.0.0, no breaking changes will occur between major releases), so you should be protected against future breaking changes by pinning your requirements with kedro-mlflow>=0.8.0, <0.9.0