38

I'm using poetry for my Python package manager but I believe this would apply to any programming practices.

I've been doing this without knowing exactly what I'm doing, or how I should be doing.

When you use a package manager and install a new package, there's usually a .lock file change to keep your build deterministic.

Usually, I would commit these changes like:

$ git add poetry.lock pyproject.toml 
$ git commit -m "Install packages: beautifulsoup4"

i.e, I make a commit every time I install/remove a package. I do it because I FEEL like this is what I should do, but I have 0 clue if this is actually a correct way to handle this.

Am I doing great? or is there any other specific convention & rules I should abide by to make it follow the best practices as close as possible?

user8491363
  • 2,924
  • 5
  • 19
  • 28
  • 1
    It's usually a good idea, especially when you are working with other developers because you will all be using the same versions of each package. It cuts down on issues where one developer has a slightly different version of a package and it is causing issues – Iain Shelvington Apr 05 '20 at 03:30
  • 3
    This isn't a full answer because I don't have intelligent thoughts about it right now, but Poetry's docs [recommend](https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control) committing `poetry.lock` to version control. But then, they also say it's "not necessary" for libraries (opposed to applications). ¯\\_(ツ)\_/¯ – Maxpm Jan 12 '21 at 23:47
  • @user8491363, would you mind unaccepting my answer? there is an authorative post from one of poetry's project maintainers, and unless I come up with an actual counter-example to `Poetry's lock file is an universal lock file`, I feel uncomfortable with the check-mark. – Arne Oct 31 '22 at 10:04

5 Answers5

32

Disclaimer

Please refer to this answer for the official stance and justification on the topic, which should be the top post in this thread. Below is my original answer, which I'll leave as it was.

The official recommendation of the poetry maintainers is to commit the lockfile if you develop a deployable application (as opposed to a library).

That being said, my personal experience has been that it isn't necessary to commit lockfiles to VCS. The pyproject.toml file is the reference for correct build instructions, and the lockfile is the reference for a single successful deployment. Now, I don't know what the spec for poetry.lock is, but I had them backfire on me often enough during collaboration with colleagues in ways where only deleting them would fix the problem.

A usual problem was that using different operation systems or python versions would lead to different lockfiles, and that just doesn't fly. I'll gladly let our CI build and persist an authoritative reference-lockfile to enable re-builds, but it still wouldn't be committed to the repository.

If maintaining a shared lockfile is viable given your workflow - great! You avoid a step in your pipeline, have one less artifact to worry about, and cut down dramatically on build time (even a medium-size project can take minutes to do a full dependency-resolution).

But as far as best practices go, I'd consider adding poetry.lock to the .gitignore a better practice than what you do, and only commit pyproject.toml changes when you add dependencies.

Arne
  • 17,706
  • 5
  • 83
  • 99
  • 1
    Sounds fair. However I'm still wondering how the majority of projects/devs are doing in the wild. What do they usually do? – user8491363 Apr 07 '20 at 09:22
  • 8
    I help maintain a number of closed and open source projects, and they all commit lockfiles, partly because I advocated in favor of it. By now I regret that choice, because it occurs quite often that someone's build is not working and the solution is to delete and re-build the lockfile, after which all of us end up having merge conflicts. – Arne Apr 07 '20 at 12:02
  • 3
    The only place where lockfiles are useful imo is after testing and verifying a build (either manually or in a CI) and saving the lockfile as a build artifact, which allows you to safely revert to older deployments. – Arne Apr 07 '20 at 12:05
  • Maybe also true for individual developers who use git only to track his/her own work, not for collaborations? – user8491363 Apr 07 '20 at 12:12
  • 4
    Even in that case it can go wrong if, for example, you want to support different platforms. If you develop on linux, and commit a lockfile that was build on said linux, then using that lockfile while e.g. your CI tries to build your project for windows will result in an error as soon as any of your dependencies has platform specific constraints. But in general, unless you actually run into such an issue (where the only solution is to stop commiting a lockfile) it is safe to include it and comes with a couple of upsides. – Arne Apr 07 '20 at 12:19
  • 2
    I think this is probably bad advice. See Rudd's comment below. – Brian Apr 07 '21 at 13:11
  • 1
    You are welcome to go ahead and commit lockfiles during development, if it works out for you then that's great. But I'll continue to let my CI generate it and store it as a build artifact that is not checked into VCS, because that's where my deployments get triggered from, not my workstation. Since it works better for me and our team, I wouldn't call it bad advice. – Arne Apr 09 '21 at 11:20
  • 1
    @Brian I added some more information regarding this topic to my main post. I still didn't change my mind, but added reasons as to why. – Arne Apr 09 '21 at 20:56
  • @Arne Fair enough! In this case, I'd highly recommend that you fully specify version numbers instead of using semantic version ranges. This is because a minor update anywhere in the dependency graph could result in large changes to what versions are actually used from build to build. This could make tracking down issues more challenging than need be. – Brian May 12 '21 at 18:21
  • 2
    Because of the [developers' reaction](https://github.com/python-poetry/poetry/issues/7211) to the broken backward compatibility, the opinion regarding adding `poetry.lock` to `.gitignore` has a substantial reason. – Felixoid Apr 03 '23 at 12:44
16

Until now, all of the given answers missed one important difference between Poetry's lock file and other tools like pip freeze: Poetry's lock file is an universal lock file.

This means that Poetry doesn't care about the current environment, neither the Python version in use, nor the platform. Instead it makes sure that dependencies are resolvable within the given Python version range in pyproject.toml. This results in a lock file that is valid on any platform with a Python version within the range given in the pyproject.toml.

This difference to other tools, that produces lock file, is also the reason why Poetry is slower in resolving dependencies. This is also the reason why it is recommended to check in the poetry.lock in your vcs. Doing so, it speed up setting up your development environment and you make sure your environment is reproducible.

finswimmer
  • 10,896
  • 3
  • 34
  • 44
  • 3
    As a state of three months later, the maintainers invalidated the idea of having `poetry.lock` in the CVS, see https://github.com/python-poetry/poetry/issues/7211. The compatibility of the file is broken between minor versions, so it just breaks the world. – Felixoid May 06 '23 at 16:59
9

Poetry actually has a section about this on their site: https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control

They sugest that it's important to actually commit this file, since that will show which versions of the libraries included were used at the time of the commit.

Committing this file to VC is important because it will cause anyone who sets up the project to use the exact same versions of the dependencies that you are using. Your CI server, production machines, other developers in your team, everything and everyone runs on the same dependencies, which mitigates the potential for bugs affecting only some parts of the deployments. Even if you develop alone, in six months when reinstalling the project you can feel confident the dependencies installed are still working even if your dependencies released many new versions since then. (See note below about using the update command.)

For libraries it is not necessary to commit the lock file.

Ruud van den Boomen
  • 599
  • 4
  • 10
2

As the state of January 2023, poetry has broken the backward compatibility between older minor versions after at least 1.3.1, maybe 1.3.0 https://github.com/python-poetry/poetry/issues/7211

So, it looks like the only way to go now is adding poetry.lock to .gitignore

Felixoid
  • 431
  • 4
  • 10
  • 2
    I wouldn't draw the conclusion from that issue that you should add `poetry.lock` to `.gitignore`. I would instead look for ways to ensure that all contributors are using Poetry versions that generate lockfiles that are compatible with each other. For example, have everyone pin their Poetry version to the same thing. Or just have everybody frequently update their Poetry version to latest. – Maxpm Jan 18 '23 at 23:17
  • 2
    I believe, it's a much more complex solution. Like an order of magnitude. My experience speaks against possible homogenous dev env for python. It looks like poetry was a kinda solution to the issue, before they disrespect semver. Now you what, need a zoo of venv with different poetry versions to manage venv inside poetry? Sounds rediculous to me. If something is broken across official repos of other Linux distros, it's considered as completely broken. – Felixoid Jan 19 '23 at 04:48
  • Ironically, we now have to manage a variety of versions of poetry, which was supposed to be the solution to version management. The poetry developers suggest pipx for that purpose. – chrishmorris Aug 25 '23 at 12:13
  • Nonsense is the least rude word I'd use to describe their decision. Other thoughts could cause sanctions to my account. – Felixoid Aug 26 '23 at 14:48
0

According to the maintainers,

Commit your poetry.lock file to version control. Committing this file to VC is important because it will cause anyone who sets up the project to use the exact same versions of the dependencies that you are using. Your CI server, production machines, other developers in your team, everything and everyone runs on the same dependencies, which mitigates the potential for bugs affecting only some parts of the deployments. Even if you develop alone, in six months when reinstalling the project you can feel confident the dependencies installed are still working even if your dependencies released many new versions since then... For libraries, it is not necessary to commit the lock file.

However, if your team uses different operating systems, hardware/CPU types etc, and you do not develop with container tech such as Docker. I would advise against committing the lock file as that would cause a lot of problem for the team. It is fine if your team builds with Docker. For example, if your lock file contains information about a library built specifically for Linux, it becomes a problem [to install from the lock file] on Windows.

Chuma Umenze
  • 933
  • 12
  • 18