10

I want to write a script to automatically setup a brand new ubuntu installation and install a django-based app. Since the script will be run on a new server, the Python script needs to automatically install some required modules.

Here is the script.

#!/usr/bin/env python

import subprocess
import os
import sys

def pip_install(mod):
    print subprocess.check_output("pip install %s" % mod, shell=True)

if __name__ == "__main__":
    if os.getuid() != 0:
        print "Sorry, you need to run the script as root."
        sys.exit()

    try:
        import pexpect
    except:
        pip_install('pexpect') 
        import pexpect        

    # More code here...

The installation of pexpect is success, however the next line import pexpect is failed. I think its because at runtime the code doesn't aware about the newly installed pexpect.

How to install and import Python modules at runtime? I'm open to another approaches.

Trenton McKinney
  • 56,955
  • 33
  • 144
  • 158
flowfree
  • 16,356
  • 12
  • 52
  • 76

6 Answers6

13

You can import pip instead of using subprocess:

import pip

def install(package):
    pip.main(['install', package])

# Example
if __name__ == '__main__':
    try:
        import pexpect
    except ImportError:
        install('pexpect')
        import pexpect

Another take:

import pip

def import_with_auto_install(package):
    try:
        return __import__(package)
    except ImportError:
        pip.main(['install', package])
    return __import__(package)

# Example
if __name__ == '__main__':
    pexpect = import_with_auto_install('pexpect')
    print(pexpect)

[edit]

You should consider using a requirements.txt along with pip. Seems like you are trying to automate deployments (and this is good!), in my tool belt I have also virtualenvwrapper, vagrant and ansible.

This is the output for me:

(test)root@vagrant:~/test# pip uninstall pexpect
Uninstalling pexpect:
  /usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect-2.4-py2.6.egg-info
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/screen.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/screen.pyc
Proceed (y/n)? y
  Successfully uninstalled pexpect
(test)root@vagrant:~/test# python test.py
Downloading/unpacking pexpect
  Downloading pexpect-2.4.tar.gz (113Kb): 113Kb downloaded
  Running setup.py egg_info for package pexpect
Installing collected packages: pexpect
  Running setup.py install for pexpect
Successfully installed pexpect
Cleaning up...
<module 'pexpect' from '/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc'>
(test)root@vagrant:~/test#
Paulo Scardine
  • 73,447
  • 11
  • 124
  • 153
  • The problem still persists: the installation is success, but the import generates `ImportError: no module named pexpect`. – flowfree Jun 21 '13 at 12:21
  • No, I write and run the script on new ubuntu installation. It only has `python-pip` package installed. – flowfree Jun 21 '13 at 12:24
  • `WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip` looks like this approach is deprecated now. See [this issue](https://github.com/pypa/pip/issues/5599) – Klesun Jul 21 '23 at 17:34
7

For those who are using pip version greater than 10.x, there is no main function for pip so the alternative approach is using import pip._internal as pip instead of import pip like :

Updated answer of Paulo

import pip._internal as pip

def install(package):
    pip.main(['install', package])

if __name__ == '__main__':
    try:
        import pexpect
    except ImportError:
        install('pexpect')
        import pexpect
pouyan
  • 3,445
  • 4
  • 26
  • 44
3

I actually made a module for this exact purpose (impstall)

It's really easy to use:

import impstall
impstall.now('pexpect')
impstall.now('wx', pipName='wxPython')

Github link for issues/contributions

ryry1985
  • 336
  • 5
  • 11
2

I solved my problem using the imp module.

#!/usr/bin/env python

import pip
import imp

def install_and_load(package):
    pip.main(['install', package])

    path = '/usr/local/lib/python2.7/dist-packages'
    if path not in sys.path:
        sys.path.append(path)

    f, fname, desc = imp.find_module(package)
    return imp.load(package, f, fname, desc)

if __name__ == "__main__":
    try:
        import pexpect
    except:
        pexpect = install_and_load('pexpect')

    # More code...

Actually the code is less than ideal, since I need to hardcode the Python module directory. But since the script is intended for a known target system, I think that is ok.

flowfree
  • 16,356
  • 12
  • 52
  • 76
2

I had the same issue but none of Google's searches helped. After hours debugging, I found that it may be because the sys.path is not reloaded with new installation directory.

In my case on my Ubuntu Docker, I want to import dns.resolver at runtime for Python3.8 (pre-installed). I also created ubuntu user and run all things with this user (including my Python script).

  • Before installing, sys.path doesn't have /home/ubuntu/.local/lib/python3.8/site-packages since I didn't install anything.
  • While installing with subprocess or pip.main like above, it creates /home/ubuntu/.local/lib/python3.8/site-packages (as user installation).
  • After installing , the sys.path should be refreshed to include this new location.

Since the sys.path is managed by site module, we should reload it (ref HERE):

    import site
    from importlib import reload
    reload(site)

The full block for anyone that needs:

import subprocess
import sys

try:
    import dns.resolver
except ImportError:
    subprocess.check_call([sys.executable, "-m", "pip", "install", "dnspython"])
    import site
    from importlib import reload
    reload(site)
    import dns.resolver

I'm not Python developer so these code can be simplify more. This may help in cases such as fresh CI/CD environment for DevOps engineers like me.

minhluantran017
  • 163
  • 6
  • Man thank you so much, I spent so much time searching on how to circumvent this issue. Everybody seemed to install and import just fine, but I was getting the ImportError like you mentioned. – rsp Mar 24 '22 at 14:31
0

I have written a very simple library that solves this problem. It's called instld.

Install it:

pip install instld

And just use in your code:

import installed

with installed('some_package'):
    import some_module

This library installs the dependencies you specified into isolated contexts right in runtime. This allows you to bypass any possible dependency conflicts, and even use 2 different versions of the same library at the same time.

And please note, the library does not leave any garbage behind. All downloaded packages are automatically deleted after exiting the context manager.

Evgeniy Blinov
  • 351
  • 3
  • 3