I started a Kedro project a while ago and started to build different parts of the pipeline which only tangentially interact with each other. In some cases not much at all.
As a consequence, as the project grew, I am starting to get issues with the dependencies given that the installation process uses requirement.txt
Thus I wanted to ask if Kedro was meant to be designed as a mono repo or not. If so, is there an example of how this could be done, please? Thanks in advance.
So Kedro isn't necessarily opinionated on the question of mono vs multi repo, but the modern way to use kedro via Modular pipelines and Micro-packaging is designed with this sort of pattern in mind.
You can even provide a module specific requirements.txt in your pipeline folder which means when users package/consume your reusable pipelines they won't take the global requirements, but just the pipeline specific ones.
Now if you are at a stage where your different pipelines are using very different requirements that are impossible to compile at a global level, it would be recommended to try and keep the requirements aligned or think about breaking it up into smaller repositories.
