How to handle development of nested graph python libraries

Question

Imagine I have:

• Library Z
• Library Y, which depends on Library Z
• Application A, which depends on Library Y

To fully test out changes to Library Z, I'd like to run the tests of Application A with any Development releases of Library Z.

To do this I can set up Library Z to publish packages to some package index for development releases under the versioning scheme {major}.{minor}.{micro}.dev{build}, then have Library Y specify it's dependency range for Library Z as >={major},<{major+1} for instance and use pip install --pre ... on Application A to ensure the Development releases of Library Z are picked up.

This all works fine, until we have > 1 maintainer of Library Z making changes, likely in different git branches, and effectively competing on the {build} number. Wondering how folks have solved this problem?

This problem gets potentially worse as well if in Application A you are also in a situation where > 1 maintainer are making changes and not everyone wants to ingest the Development release, so ensure the --pre flag is optionally passed and ideally synced up with just the dependency in question (possible with poetry via the more granular allow-prereleases flag, see docs here).

---

Editable installs are likely considered out of scope, this set up is a trivial case. In reality this dependency graph could be deeper, and is often pared with Docker to make it commercially viable when pared with C dependencies so the complexity of hooking up volume mounts very hard. Also the user developing Library Z, may be different than the person testing Application A.

Whilst I used pip in the examples here, in reality our system uses poetry (and pip in places).

Answer 1

[given the information exchanged in the comments...] Then version control branches should do. If you agree that a certain branch in lib Z will have the new features app A needs for the purposes of build W, just keeping that branch updated won´t require any changes on the configuration of W, and no package building: at this point you are down to being able to switch the requirement list (and the pointers to each branch) on the build process of A.

I don't know how Poetry could handle this, but using a setup.py file it should be trivial: just read different text files listing the requirements (with pointers to specific GIT branches or Tags), based on a system-variable or other configuration you can easily change for the build. Since setup.py are plain Python code file, one can just use if statements along with os.environ to check environment variables, and read the contents of requirements_setup_W.txt to feed the install_requires parameter of the call to setup.

Either way, you are saying each package might have more than one state of "pre-release", which would be interesting for different builds - I think managing that through branches in the version control would be better than uploading several differing packages for a repository. or maybe, change the uploaded package name for each finality. So, you could have a package named libZ_setupW on the local pypi (and rotate the requirements by code on setup.py . (again, the main pain point of poetry is trashing executable code expecting all build needs can be represented by config files: they can't. I just looked around, and there seems to be no "poetry pre-build hook", which could be used to rename (or dynamically rewrite) your pyproject.toml according to the desired setup. One could add a script to do just that, but it would need to be called manually before calling poetry install)