3

I have a repository with multiple Python projects that use Poetry for dependency management. The directory structure is:

  • proj1

    • src/
    • pyproject.toml

  • proj2

    • src/
    • pyproject.toml

  • common_proj

    • src/
    • pyproject.toml

The proj1 and proj2 projects both depend on common_proj. Currently, I'm building proj1 and proj2 like this:

cd proj1
cp -r ../common_proj/src ./common_proj
poetry build

Then I use the generated wheel file in production.

The pyproject.toml files for proj1 and proj2 include common_proj like this:

packages = [
    {include = "proj2"},
    {include = "common_proj" },
]

What is the correct way to manage the common_proj dependency with Poetry, so that its dependencies are installed properly?"

Ofer Helman
  • 714
  • 1
  • 8
  • 25
  • 1
    To manage the common_proj dependency properly with Poetry in your multiple Python projects (proj1 and proj2), you should create a dedicated directory for your common_proj package and publish it as a separate package. This way, you can version and manage the common_proj independently, and proj1 and proj2 can reference it as a dependency in their respective pyproject.toml files. – Prashant Patil Jul 28 '23 at 12:22

2 Answers2

2

I have recently been dealing with a similar problem to this. I have a couple of solutions to this and myself have implemented some of these for different problems depending on the use case.

1. Publish common_proj as its own package

If you publish common_proj as its own package you can then install this in your other packages. I believe this is probably the nicest approach. It also decouples the different packages a little more which in the long run might be of much more value.

2. Use a script for building and publishing

This is a bit more involved than option 1. However, I have found that this one works for cases where I am told I am absolutely not allowed to publish the common_proj as a package. The script here is pretty much exactly as you described above. In usecases where you have multiple different common_proj and some subset of them that might be used then it can get more complicated.

3. The docker approach

This is a similar approach to the one above. However, I find it works slightly nicer assuming you will already be packaging the projects up into dockerfiles. I have used this a couple of times where all the projects get deployed to serverless functions. With them all using similar classes that sit in the shared folder.

Rather than having multiple different projects all the code is in 1 project as a mono-repo.

project
-- proj_1
   -- files.py
-- proj_2
   -- files.py
-- common_proj
   -- files.py
pyproject.toml
poetry.lock

The dockerfile can be build based on which project you are trying to build. You then have a script you run that will read the pyproject.toml and remove the libraries for projects not currently being built. This works best when the projects are very similar often with the same required packages but need to be deployed separately.

Conclusion

Publishing the common_proj as a package is the nicest, it makes life the easiest. The other solutions have trade-offs and rely on adding extra complexity to your projects to manage packages. That said this can all be automated with scripts that you run instead of those you are used to. I would really recommend against them unless you absolutely cannot publish the package.

Edited 2023-07-09 to make it clear when the docker solution makes sense.

Karrot96
  • 91
  • 6
  • You definitely don't need a Docker setup to create a monorepo. – Original BBQ Sauce Jul 27 '23 at 11:00
  • @OriginalBBQSauce I should of probably been a bit more clear on when the docker approach makes the most sense. In the cases I have used this approach, the projects are deploying functions to serverless infrastructure. So the docker files are existing anyway. When I have some time will edit the above to make this clear – Karrot96 Jul 28 '23 at 12:45
1

If you want to clearly separate the projects, then publish them. No question about it.

By "publishing it" I don't mean to create a wheel and push it to PyPi. You can get away with a lot by just having it as a separate package, with its own pyproject.toml file, and listing it as a git dependency on your other projects (e.g.: poetry add git+https://github.com/username/common_proj.git).

At the end of the day, your question is really about having multi-repos vs a single monorepo. And that's your other option: to treat everything as a single project.

Then proj_1, proj_2, and common_proj all become submodules. It doesn't look like the path you want, but honestly, there's nothing inherently wrong with it and monorepos make it easier to move stuff between projects.

If you divide the source code into separate repositories before getting the boundaries right, you'll build a tremendous amount of friction into your system.

Original BBQ Sauce
  • 527
  • 1
  • 11
  • 24