shebang: use interpreter relative to the script path

Question

I try to build scripts that work everywhere and always. For this I use a custom-built python, which is always in the parent directory relative to the script.

This way I could load my package on an USB-stick and it would work everywhere, regardless of where the stick is mounted and whether python is installed or not.

However, when I use

#!../python

then it works only when the script gets invoked from its directory, which is of course not acceptable.

Is there a way to do this or is this impossible in the current shebang-mechanism?

Answer 1

There is a healthy set of multi-line shebang scripts on this page for a lot of languages, example:

#!/bin/sh
"exec" "`dirname $0`/python" "$0" "$@"
print copyright

And if you want one-line shebang, this answer (and question) explains the issue in the details and suggests the following approaches using additional scripts inside the shebang:

Using AWK

#!/usr/bin/awk BEGIN{a=ARGV[1];sub(/[a-z_.]+$/,"python",a);system(a"\t"ARGV[1])}

Using Perl

#!/usr/bin/perl -e$_=$ARGV[0];exec(s/\w+$/python/r,$_)

update from 11Jan21:

Using updated env utility:

$ env --version | grep env
env (GNU coreutils) 8.30
$ env --help
Usage: env [OPTION]... [-] [NAME=VALUE]... [COMMAND [ARG]...]
Set each NAME to VALUE in the environment and run COMMAND.

Mandatory arguments to long options are mandatory for short options too.
  -i, --ignore-environment  start with an empty environment
  -0, --null           end each output line with NUL, not newline
  -u, --unset=NAME     remove variable from the environment
  -C, --chdir=DIR      change working directory to DIR
  -S, --split-string=S  process and split S into separate arguments;
                        used to pass multiple arguments on shebang lines

So, passing -S to env will do the job now

Answer 2

Expanding @Anton's answer so as not to fail on spaces and other special characters in the path and to explain a bit more about the magic.

#!/bin/sh
"true" '''\'
exec "$(dirname "$(readlink -f "$0")")"/venv/bin/python "$0" "$@"
'''

__doc__ = """You will need to deliberately set your docstrings though"""

print("This script is interpretable by python and sh!")

This clever script is comprehendable by both sh and python. Each of which react to it differently. The first 2 lines after the shebang are interpreted by sh and cause it to hand over exec to a relative python binary (supplied with the same command line arguments). These same lines are safely discarded by python since they amount to a string ("true") followed by a multi-line string (''').

A little more about this subject can be read here.

Answer 3

After looking at these answers, I decided to use a Python-specific solution that does not actually change the shebang.

This solution uses the system python interpreter to find the desired interpreter and execute that. This is useful to me because it allows me to munge environment variables as well as ensuring the correct interpreter.

Because it uses exec, it does not double the memory use-- the new interpreter replaces the last one. Also, exit status and signals will be handled correctly.

It is a python module that is loaded by the script that needs to run under this environment. Importing this module has the side-effect of launching a new interpreter if needed. Generally, modules should not have side-effects, but the alternative is to run the module's function before non-system imports are performed, and that would be a PEP8 violation. So you have to pick your poison.

"""Ensure that the desired python environment is running."""


import os
import sys


def ensure_interpreter():
    """Ensure we are running under the correct interpreter and environ."""
    abs_dir = os.path.dirname(os.path.abspath(__file__))
    project_root = os.path.normpath(os.path.join(abs_dir, '../../../'))
    desired_interpreter = os.path.join(project_root, 'bin/python')

    if os.path.abspath(sys.executable) == desired_interpreter:
        return

    env = dict(os.environ)

    def prefix_paths(key, prefix_paths):
        """Add prefix paths, relative to the project root."""
        new_paths = [os.path.join(project_root, p) for p in prefix_paths]
        new_paths.extend(env.get(key, '').split(':'))
        env[key] = ':'.join(new_paths)

    prefix_paths('PYTHONPATH', ['dir1', 'dir2'])
    prefix_paths('LD_LIBRARY_PATH', ['lib'])
    prefix_paths('PYTHON_EGG_CACHE', ['var/.python-eggs'])
    env['PROJECT_ROOT'] = project_root
    os.execvpe(desired_interpreter, [desired_interpreter] + sys.argv, env)


ensure_interpreter()

If you don't need to munge any environment variables, you can strip out everything between env = dict(os.environ) and os.execvpe(desired_interpreter, [desired_interpreter] + sys.argv, env).

Answer 4

Here is a shebang that will pass the arguments, work with scripts that have non-word characters in the name, and work with older versions of perl as well (the non-destructive substitution flag /r was not added until 5.13.2). The perl example in the accepted answer only passes the script itself as an argument instead of all the arguments with @ARGV.

#!/usr/bin/perl -e$_=$ARGV[0];s/[^\/]+$/python/;exec($_,@ARGV)

This shebang runs perl to run a inline script -e. It assigns the path of the script $ARGV[0] to the default variable $_. The perl substitution s/[^\/]+$/python/ is then performed on the default variable to replace all non forward slash / characters ([^\/]+) at the end of the path (anchor to end $) with the name of the alternate command to run (python). Finally it runs the new command with exec($_ and passes all the arguments of the original call to it (@ARGV)

Answer 5

For others who find this question while looking for a portable solution to the python hashbang line, and find the AWK command above does not work with passing multiple arguments, use the following instead.

#!/usr/bin/awk BEGIN{a=ARGV[1];b="";for(i=1;i<ARGC;i++){b=b" \""ARGV[i]"\"";}sub(/[a-z_.\-]+$/,"python",a);system(a"\t"b)}

To change all the scripts hashbang lines in the current directory you can execute the following.

sed -i '1 s|^#!.*|#!/usr/bin/awk BEGIN{a=ARGV[1];b="";for(i=1;i<ARGC;i++){b=b" \""ARGV[i]"\"";}sub(/[a-z_.\-]+$/,"python3.5",a);system(a"\\t"b)}|' *

Answer 6

(Can't comment due to missing credit)

Re bdrx's answer (https://stackoverflow.com/a/62268465/7379507): Didn't work for me, unfortunately. Looks like @ARGV is the culprit here, leaving it out does work (Perl v5.16.3):

#!/usr/bin/perl -e$_=$ARGV[0];s/[^\/]+$/python/;exec("'$_'")                  

Without additional args, of course. The single quotes are necessary to safeguard for paths with spaces in directory or filenames.

Re Kenneth E. Bellock's answer (https://stackoverflow.com/a/36160331/7379507) and clarifying comments: This also has issues with blanks in path parts.

This seems to (mostly) work for me (GNU Awk 4.0.2):

(SEE BELOW for a better variant with proper return code - I'm leaving this in for educational purposes)

#!/usr/bin/awk BEGIN{a=ARGV[1];b="";for(i=2;i<ARGC;i++){b=b" "ARGV[i];}sub(/[^\/]+$/,"python",a);system(sprintf("'%s'%s",a,b))}                               

Modifications:

• Regex substitutes everything but forward slashes
• quoting safeguards for dir and filenames with blanks
• simple additional args like -v -i seem to work, but not quoted args like -c'import sys'
• I couldn't quite grasp the \t stuff, so start the loop with i=2 and combine ARGV entries with blanks

Thanks for the inspiration. Still, quoting on the shebang remains some sort of mystery for me... ;-)

EDIT: Just noticed that I accidently dropped the "rc=system(..);exit rc" part so lose the Python process return code. And I can't get it work my modifications. Yikes!

EDIT2: Ok, the problem with the return code lies in the shebang line-length restriction. But I realized I can do without sprintf():

!/usr/bin/awk BEGIN{a=ARGV[1];b="";for(i=2;i<ARGC;i++){b=b" "ARGV[i];}sub(/[^\/]+$/,"python",a);rc=system("'"a"'"b);exit rc}

Still, the problem with quoted args remains.

Answer 7

I suggest this, following this answer: https://stackoverflow.com/a/73696739/343036 :

#!/usr/bin/env -S /bin/sh -c '"$(dirname "$0")/python" "$0" "$@"'

It requires a recent-enough linux version - Ubuntu 20.04 supports it, 18.04 doesn't.

The reason I like this: it is quite readable, it doesn't matter if you use Python or another interpreter, and it manages extra arguments as expected.