How to create requirements.txt in Python with actually used libraries?

Question

I find Colab pre-installed libraries convenient as well as AWS Sagemaker pre-defined kernels but this convenience becomes very annoying when gathering the requirements.txt file as I end up with many libraries I have not actually used in my project. I know I could create a virtual at the very beginning but I am wondering if there is a way to avoid it.

I have recently discovered watermark which partially solves this issue. Nevertheless, for this solution to be a perfect fit it still has two issues that I will exemplify below and that you can easily reproduce in Colab.

!pip install fastai --upgrade
!pip install voila
!jupyter serverextension enable voila --sys-prefix
!pip install watermark
from fastai.vision.all import *
from fastai.vision.widgets import *
%load_ext watermark
%watermark --iversion

Neither fastai nor voila appear in the output as I am not running import fastai and loading voila as an extension.

%watermark -p fastai

This would return the correct output for e.g. fastai but I would like to be able to generate automatically without having to manually check for the missing packages.

score 0 · Answer 1 · edited Jan 11 '21 at 21:02

While this isn't IPython magic, I wrote this little python script (which you could copypasta as a cell and run).

It works by looking at the imported modules, cross-referencing them against those in your pip installed distributions, then creates a requirements.txt off of that information.

from pip._internal.utils.misc import get_installed_distributions
import sys
#import numpy as np # imported to test whether numpy shows up, which it does!

def get_imported_packages():
    p = get_installed_distributions()
    p = {package.key:package.version for package in p}

    imported_modules = set(sys.modules.keys())
    
    imported_modules.remove('pip')

    modules = [(m, p[m]) for m in imported_modules if p.get(m, False)]

    return modules


def generate_requirements(filepath:str, modules):
    with open(filepath, 'w') as f:
        for module, version in modules:
            f.write(f"{module}=={version}")


generate_requirements('requirements.txt', get_imported_packages())

I run the code but still do not see "voila" in the requirements.txt. Only fastai. — G. Macia, Jan 15 '21 at 07:55
I also would suggest adding a new line character here f.write(f"{module}=={version}\n") — G. Macia, Jan 15 '21 at 07:55

How to create requirements.txt in Python with actually used libraries?

1 Answers1