How to update a requirements file automatically when new packages are installed?

Question

Keeping track of a virtual environment's requirements via pip freeze is simple.

pip freeze > requirements.txt

Currently, however, whenever a new package is added to the venv, it needs to be added to the requirements file manually. To do so, I usually just run the freeze command again and pipe it into the requirements file, but sometimes I forget to run this command, and this can be troublesome especially in repositories across different locations whenever I have to remember which packages I need to install!

Whenever a new package is installed in a virtual environment, is there any way to automatically update a requirements.txt file automatically to include this new package?

Answer 1

When using just plain pip to install packages, there is currently no way to make it automatically generate or update a requirements.txt file. It is still a manual process using pip freeze > requirements.txt.

If the purpose is to make sure the installed packages are tracked or registered properly (i.e. tracked in version control for a repository), then you will have to use other tools that "wrap" around pip's functionality.

You have two options.

Option 1: Use a package manager

There are a number of Python package managers that combines "install package" with "record installed packages somewhere".

• pipenv
  • "It automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages. It also generates the ever-important Pipfile.lock, which is used to produce deterministic builds.
  • Workflow (see Example Pipenv Workflow)

    $ pipenv install some-package
    
    $ cat Pipfile
    ...
    [packages]
    some-package = "*"
    
    # Commit modified Pipfile and Pipfile.lock
    $ git add Pipfile*
    
    # On some other copy of the repo, install stuff from Pipfile
    $ pipenv install
    

• poetry
  • "poetry is a tool to handle dependency installation as well as building and packaging of Python packages. It only needs one file to do all of that: the new, standardized pyproject.toml. In other words, poetry uses pyproject.toml to replace setup.py, requirements.txt, setup.cfg, MANIFEST.in and the newly added Pipfile.*"
  • Workflow (see Basic Usage)

    $ poetry add requests
    
    $ cat pyproject.toml
    ...
    [tool.poetry.dependencies]
    requests = "*"
    
    # Commit modified pyproject.toml
    $ git add pyproject.toml
    
    # On some other copy of the repo, install stuff from Pipfile
    $ poetry install
    

Option 2: git pre-commit hook

This solution isn't going to happen "during installation of the package", but if the purpose is to make sure your tracked "requirements.txt" is synchronized with your virtual environment, then you can add a git pre-commit hook that:

1. Generates a separate requirements_check.txt file
2. Compares requirements_check.txt to your requirements.txt
3. Aborts your commit if there are differences

Example .git/hooks/pre-commit:

#!/usr/local/bin/bash

pip freeze > requirements_check.txt
cmp --silent requirements_check.txt requirements.txt

if [ $? -gt 0 ]; 
then
  echo "There are packages in the env not in requirements.txt"
  echo "Aborting commit"
  rm requirements_check.txt
  exit 1
fi

rm requirements_check.txt
exit 0

Output:

$ git status
...
nothing to commit, working tree clean

$ pip install pydantic

$ git add .
$ git commit
The output of pip freeze is different from requirements.txt
Aborting commit

$ pip freeze > requirements.txt
$ git add .
$ git commit -m "Update requirements.txt"
[master 313b685] Update requirements.txt
 1 file changed, 1 insertion(+)

Answer 2

Instead of pip, use pipenv. It is a much better dependency manager, that will ensure best practices and remove manual work.

To learn the use of pipenv read this article.