Mess with venv or new flavor of python?

Question

Question

The question is if there another way that I'm just not thinking of to solve those problems below, or is it really the answer to build a python flavor with our tools? I have a proposed solution that solves the problems, but that doesn't mean it's the right answer.

Problems

I work in a support organization where we're developing tools for support on our main product. This product has it's own flavor of OS that it runs on. We have three distinct problems to solve in packaging o

1. The flavor of OS has it's original python binaries we can install to, but this limits us to the OS version of python which will change as a separate team that manages the OS. This is expected to change over the next 2 years between releases 3.4, 3.5, and 3.6 of python - which have changes that affect libraries we use consistently, for better and worse.

2. The tools that we build are also used as standalone tools at sites with no connectivity externally for analysis. We're currently limited to "hope the python that's on wherever they're using it plays nice".

3. There are significant performance improvements in later versions of python that we would like to take advantage of, and can't without upgrading to 3.6, but that removes our ability to use the older versions because of some significant differences in the python lib that break things in one vs. the other.

My original approach was to try and make a standalone virtualenv that was relocatable, but the more I looked at that code, the more I found that it's just editing PYTHONHOME and PATH, and if you wanted it to be relocatable, you have to copy all the binaries anyway, or it's only usable if the host has the version of python you're built for. This also had the downside of needing a lot of modification to the virtualenv scripting so that the shebangs were modified, the paths were updated, etc - and would need to be updated on every move, or have dynamic shebangs.

Proposed Solution that feels wrong

Right now I'm looking at creating our own "flavor" of python - but that feels like bringing an axe to a dinner party to cut your carrots. It solves all of the problems of multiple locations that it could be used all having a consistent up to date version of python with the tools we've installed, so all the users would need to do is run a script that updates the PATH with the /bin where these things are installed.

So back to the question:

Is there another way that I'm just not thinking of to solve those problems, or is it really the answer to build a python flavor with our tools? Am I making this feel wrong because of inexperience, or is this a valid answer I should be considering?

Answer 1

You can actually install Python locally, to a directory without root privileges, and run it completely standalone from any system Python. Your tool can use this standalone Python install instead of any sort of system-level dependency.

• Unzip the Python release using tar -xzf Python-X.X.X.tgz
• Make a directory for the installed Python using mkdir ~/python36
• Enter the unzipped install file directory using cd Python-X.X.X
• Generate the Python make-file for your system using ./configure -prefix=/home/user/python36
• Install Python to the local directory using make altinstall prefix=/home/user/python36 exec-prefix=/home/user/python36 (Note that the use of altinstall is critical, as this ensures the Linux dependency Python 2.7 or other installed version is not replaced)

Now, you can run your project on this local install using /home/user/python36/bin/python3 script.py Using this method, you can develop with one version of Python in mind, and not have to anticipate changes that you aren't able to control. If you want to update your Python for your tool, it is a very intentional and atomic action.

Answer 2

Similar to J. Blackadar answer, I know at least one company who does that.

Heroku, a cloud provider, has a CLI tool that is written in NodeJS. Heroku uses homebrew to distribute the package. They also use the same package manager to distribute their own version of nodejs, named heroku/brew/node. You can see that sometimes the program itself gets updated, and when there is an update in NodeJS they update both. The catch is ,sometimes NodeJS itself gets updated but heroku doesn’t update it’s own NodeJS , so they can keep going using their own version until they run the proper tests with the new version.

You can also install your own version of python and update it separately or use virtualenv to install specific version of python, but I believe you already tried it.

Another option is to use docker of course, you will have the same environment everywhere, but that might add more complexity now that you have a special OS.

Answer 3

I actually ended up blending the two - I created my own "activate" file that dynamically allocates path (To whatever the activate file has been called fromhas some additional functionality, and put that into a python binary. When sourced, it will update the shebangs.

1. activate will now set the path dynamically, using it's own directory that it resides in as the new PATH shim. This assumes, as with most venvs, that the activate script resides in bin.

2. activate now has a helper script and a wrapper that calls it for all files in bin that have a shebang. It updates those files shebang to the path of the current python executable (Which should have been updated by the virtualenv activate script at this point). It will only do this if the shebangs are different.

The script changes for reference:

function update_shebang() {
    # Call like: update_shebang check_sas_cabling
    full_path=${1}
    # Back up our scripts, making them hidden by default.
    file_dir=$(dirname $full_path)
    file_name=$(basename $full_path)
    cp ${full_path} "${file_dir}/.${file_name}.bak"
    # Find old shebang - will look something like this:
    # /workdir/pure_python/bin/python3
    oldbang=$(grep '#!' ${full_path})
    newbang_path=$(readlink -f `which python`)
    # We have to escape the !, but then it keeps the backslash, so we have to
    # get rid of it.
    newbang=$(echo "#\!$newbang_path" | tr -d '\\')
    # Just using sed in place
    # We dont' want to spam people every time they activate - so only modify
    # them if they aren't the same.
    if [ "$(echo $oldbang | tr -d "/")" != "$(echo $newbang | tr -d "/")" ]; then
        # Don't modify binary matches.
        if [ ! "$(echo $oldbang | grep 'Binary')" ]; then
            echo "Updated shebang for ${file_name} from ${oldbang} to ${newbang}"
            sed -i "s|${oldbang}|${newbang}|" ${full_path}
        fi
    fi
}

function update_all_shebangs() {
    # Wrapper around update_shebang to update all the
    # shebangs in ${NEW_PATH}/bin.
    given_path=$1
    myfiles=$(ls $given_path | grep -ve "^activate$")
    for file in $myfiles; do
        fullpath=$given_path/$file
        if grep -q "#!" $fullpath; then
            update_shebang $fullpath || exit 1
        fi
    done
}

And the new dynamic path portion:

NEW_PATH="$(dirname $(dirname $(readlink -f -n $BASH_SOURCE)))"

I also incorporated creating this in a docker image that ran all the pip installs for my required stuff and then bundled it with all of this - so it takes a little bit longer, but is truly relocatable in all senses of the word right out of the box.

Will put up the skeleton repo that includes this once I script the docker installation that's required. :D