4

I looking for a way to audit and minimize requirements.txt. I've taken over a project that has grown bloated over several iterations, and I'm trying to make it more maintainable. The current virtual environment I'm in was created from the previous requirements.txt; that file has packages that are no longer imported in any script.

In the past, I've done this manual process:

  1. Search through project directory to find all python files in all subfolders
  2. Search through each python file found to find all import x and from x import y statements, add those packages to a list
  3. pip show each of packages on the list, adding any dependencies to the end of the list
  4. Once list is exhausted, sort and compare to requirements.txt
  5. Remove requirements that aren't on the list.

Assuming that my code performs no relative imports, is there a way to automate this process? I can't imagine I'd be the first person looking for such a tool (or gist, or script). I couldn't find any. I use windows, but I'm happy to linux commands on windows subsystem for linux.

sinoroc
  • 18,409
  • 2
  • 39
  • 70
Jake Stevens-Haas
  • 1,186
  • 2
  • 14
  • 26
  • 1
    I usally just delete the venv and requirements.txt, try running and install packages as long as I get import errors, then after everything works just pip freeze the requirements again :D – ruohola Aug 02 '19 at 20:23
  • 2
    @ruhola I've asked around the office, and that's the consensus lol. – Jake Stevens-Haas Aug 02 '19 at 20:43
  • Write unit tests and use some tool like `tox` to run them regularly in different environments. You will be immediately notified if the `requirements.txt` gets outdated. – hoefling Aug 02 '19 at 21:04
  • @hoefling, does `tox` tell you when requirements.txt has unneccessary packages, or when it's missing packages? The documentation seems to indicate the latter, but I'm looking for the former. – Jake Stevens-Haas Aug 02 '19 at 21:13
  • 1
    This is indeed an interesting question. No, it will not notify you on unused requirements, for example if you removed dependent code in a refactoring cycle. – hoefling Aug 02 '19 at 21:18
  • [Poetry](https://github.com/sdispater/poetry) gets you all of this stuff, but you won't be using `requirements.txt` at all, instead you will have a poetry lockfile. – wim Aug 02 '19 at 22:13
  • @wim can you explain how as an answer? The documentation for poetry doesn't seem to describe this. – Jake Stevens-Haas Aug 03 '19 at 00:02

3 Answers3

2

Maybe creosote can help in this use case. Otherwise there is pigar and pipreqs.

sinoroc
  • 18,409
  • 2
  • 39
  • 70
0

I think a tool that would help with would be pip_missing_reqs.

maciek
  • 3,198
  • 2
  • 26
  • 33
0

https://pypi.org/project/pipdeptree/ will print out a dependency graph. You can then compare top-level dependencies with imports in your code.

https://pypi.org/project/findimports/ will find imports by analyzing source files.

It would be nice if someone made an automated tool.

Nick Groesz
  • 1
  • 1