pip install with random temporary directory

Question

when i use "pip install" to install packages, the path of random temporary directory will be recorded in pyc file, How can I eliminate this random difference？(as following 2b2_9obo)

# uncompyle6 tools.cpython-37.pyc | head -n6

# uncompyle6 version 3.6.4
# Python bytecode 3.7 (3394)
# Decompiled from: Python 3.7.3 (default, Mar  6 2020, 14:23:55)
# [GCC 4.8.5]
# Embedded file name: /tmp/pip-unpacked-wheel-2b2_9obo/diffoscope/tools.py
# Size of source mod 2**32: 4106 bytes

What for? Random parts or not that's the path that was used. — Marcin Orlowski, Mar 18 '20 at 09:56
Reproducible builds needs to eliminate this difference，how can i do? — tongyishu, Mar 18 '20 at 10:05
@tongyishu https://github.com/pypa/pip/issues/5648 was created in 2018, still open. — phd, Mar 18 '20 at 12:22
This is a pip 20 regression https://github.com/pypa/pip/issues/7808. — FRidh, Jun 22 '20 at 11:56

score 1 · Answer 1 · answered Aug 16 '20 at 13:35

You can download the package first and then compile it manually:

pip install tools
python -m compileall -d nameofchoice

However, this will not make your build reproducible. The .pyc header will still contain a timestamp. As of python 3.7, you can compile it with a non-default invalidation mode.

pip install tools
python -m compileall --invalidation-mode unchecked-hash -d nameofchoice

This will make probably make your .pyc file reproducible.

rocky · Answer 2 · 2020-04-07T15:38:32.717

There's probably a way to use plain old POSIX sed on the bytecode and that will work assuming the filenames don't appear in the code other than in the co_filename field of the code object.

However if you want a more bullet-proof solution, here is how you might modify the bytecode and write that back out.

#!/usr/bin python
"""
Renames the co_filename object
"""
import glob, re, types
import os.path as osp
from xasm.misc import write_pycfile
from xasm.assemble import Assembler
from xdis.code import iscode
from xdis.load import load_module
from collections import deque

def change_co_filename(code, new_path):
    return types.CodeType(
        code.co_argcount,
        code.co_kwonlyargcount,
        code.co_nlocals,
        code.co_stacksize,
        code.co_flags,
        code.co_code,
        code.co_consts,
        code.co_names,
        code.co_varnames,
        new_path,
        code.co_name,
        code.co_firstlineno,
        code.co_lnotab,
        code.co_freevars,
        code.co_cellvars)

def disco_loop(queue, new_path):
    new_code = change_co_filename(queue[0], new_path)
    while len(queue) > 0:
        co = queue.popleft()
        for c in co.co_consts:
            if iscode(c):
                change_co_filename(c, new_path)
                queue.append(c)
            pass
        pass
    return new_code


suffix = "cpython-3.7.pyc"
paths = glob.glob(f"*.{suffix}")


# Modify to figure out what prefix to look for
orig_dirname = "/tmp/pip-unpacked-wheel-2b2_9obo/"
new_dirname = "/tmp/"

for path in paths:
    assembler = Assembler("3.7")
    version, assembler.timestamp, magic_int, code, is_pypy, assembler.size = load_module(path)
    new_path = re.sub(orig_dirname, new_dirname, code.co_filename)
    queue = deque([code])
    new_code = disco_loop(queue, new_path)
    assembler.code_list = [new_code]
    write_pycfile(path, assembler)

This relies on my xdisand xasm programs that work on bytecode. (I just now had to modify the latter for 3.7 since the code object changed in that version)

If there is interest in this kind of stuff, I can probably modify the above as a standalone routine and distribute that with xasm.

If you want to see what's going on in writing Python bytecode files, see the answer here

fwiw, this can be done entirely with stdlib `marshal` if you're looking for a no-dependencies solution: https://github.com/Yelp/virtualenv-tools/blob/c355a0b77f77b125c58f7576e47ccff727c47103/virtualenv_tools.py#L117-L162 — anthony sottile, Apr 07 '20 at 15:42
@AnthonySottile This is cool - I see code.replace() function was added. And as I mentioned there is sed. I see that the code I have above can probably be simplified. However the problem with such code like this is that it is fragile. As we've seen, the code structure changes around a lot, and that is precisely what xdis is isolating you from. On the other hand if you don't mind keeping up to date with Python Code object changes and changing your code to accomdate, the the code in the link you give is fine. — rocky, Apr 07 '20 at 15:53

pip install with random temporary directory

2 Answers2