calling virtualenv bundled pip wuth subprocess.call fails on long path to venv directory on Linux

Question

The following line throws strange exceptions in Linux (several Ubuntu 14.04 installations and one Arch linux installation with all updates installed):

subprocess.call(['/home/vestnik/Development/test/python/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/venv/bin/pip','install','httpbin'])

python2.7:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/subprocess.py", line 523, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

python3.5:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/subprocess.py", line 557, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/home/vestnik/Development/test/python/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/venv/bin/pip'

pip executable I'm trying to run is valid existing file created by virtualenv bootstrap script. The problem appears when path length to pip becomes larger then 120 characters (approximatelly). If I remove one "subdir" element in the middle of the path everyting works well. Original source of this issue is jenkins build failing on virtualenv creation due to long workspace names.

Are there any limitations on executable path length in subprocess module (I haven't find it in the function docs) or is it an implementation bug? Are there good ways to workaround it?

I'm not familiar with path limitations, but have you checked permissions on the pip file, and is the user executing the script able to rwx it? — AbrahamB, Oct 13 '16 at 06:07
I've tried to execute pip from the generated venv in bash and find out that reall issue is hash bang line lenght limit. Python just report this error in confusing way. — VestniK, Oct 13 '16 at 06:22

score 0 · Accepted Answer · edited May 23 '17 at 10:27

When I've tried to call pip in venv from bash I got bad interpretator error.

The real source of the problem is shebang string length limitation (the comment to accepted answer here claims default limit to be equal to 127 bytes):

$ head -n1 /home/vestnik/Development/test/python/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/venv/bin/pip 
#!/home/vestnik/Development/test/python/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/venv/bin/python3.5
$ head -n1 /home/vestnik/Development/test/python/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/subdir/venv/bin/pip | wc -c
129

the problem source is not subprocess module. It only producess confusing error.

The following code works:

subprocess.call([
    '/home/...lonl long venv path.../venv/bin/python3.5',
    '/home/...lonl long venv path.../venv/bin/pip',
    'install',
    'httpbin'
])

calling virtualenv bundled pip wuth subprocess.call fails on long path to venv directory on Linux

1 Answers1