0

I'm cross-building python 3.8. Two errors look simiar:

$ python3 -m venv .
Error: Command '['/home/p3t/penv386/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

$ bin/python3 -Im ensurepip --upgrade --default-pip
Traceback (most recent call last):
  File "<frozen zipimport>", line 520, in _get_decompress_func
ModuleNotFoundError: No module named 'zlib'

$ pip3 install --user pipenv
Traceback (most recent call last):
  File "/usr/bin/pip3", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 35, in <module>
  File "/usr/lib/python3.8/plistlib.py", line 65, in <module>
ModuleNotFoundError: No module named 'xml'

I've seen a few similar threads on SO:

  • build python with zlib: Edit Module/Setup to include zlib or install zlib. I did the edit and verified libz exists. It also has a python snippet to verify but should be run if import ok.
  • no module named zlib: Similar solution.
  • A few other threads are basically very similar to the above two.

Since the "/usr/lib/libz*" does exist and the xml lib "/usr/lib/libexpat*" does exists too, I start to think the module info probably is missing. How to make sure those two modules are properly populating with their information in the python build?

I cross-compiled using buildroot 2020.08.2. From the buildroot configuration, both xml and zlib are selected. I did a strace, have not seen what could be the cause, here for the venv:

$ cat -n log2-venv | egrep '(openat|newfstatat)' | grep '\.so'
     5  5835  openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
     6  5835  openat(AT_FDCWD, "/lib64/tls/aarch64/libpython3.8.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
     8  5835  openat(AT_FDCWD, "/lib64/tls/libpython3.8.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    10  5835  openat(AT_FDCWD, "/lib64/aarch64/libpython3.8.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    12  5835  openat(AT_FDCWD, "/lib64/libpython3.8.so.1.0", O_RDONLY|O_CLOEXEC) = 3

    20  5835  openat(AT_FDCWD, "/lib64/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = 3
    28  5835  openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
    36  5835  openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
    43  5835  openat(AT_FDCWD, "/lib64/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
    50  5835  openat(AT_FDCWD, "/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
    58  5835  openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

   123  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/encodings/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad5c10, 0) = -1 ENOENT (No such file or directory)
   124  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/encodings/__init__.abi3.so", 0x7fc2ad5c10, 0) = -1 ENOENT (No such file or directory)
   125  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/encodings/__init__.so", 0x7fc2ad5c10, 0) = -1 ENOENT (No such file or directory)

   496  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/collections/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad43a0, 0) = -1 ENOENT (No such file or directory)
   497  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/collections/__init__.abi3.so", 0x7fc2ad43a0, 0) = -1 ENOENT (No such file or directory)
   498  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/collections/__init__.so", 0x7fc2ad43a0, 0) = -1 ENOENT (No such file or directory)

   544  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_heapq.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=18528, ...}, 0) = 0
   546  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_heapq.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3

   578  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/pkg_resources/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad60c0, 0) = -1 ENOENT (No such file or directory)
   579  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/pkg_resources/__init__.abi3.so", 0x7fc2ad60c0, 0) = -1 ENOENT (No such file or directory)
   580  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/pkg_resources/__init__.so", 0x7fc2ad60c0, 0) = -1 ENOENT (No such file or directory)

   615  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/binascii.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=27656, ...}, 0) = 0
   617  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/binascii.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3

   629  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/importlib/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad3af0, 0) = -1 ENOENT (No such file or directory)
   630  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/importlib/__init__.abi3.so", 0x7fc2ad3af0, 0) = -1 ENOENT (No such file or directory)
   631  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/importlib/__init__.so", 0x7fc2ad3af0, 0) = -1 ENOENT (No such file or directory)

   766  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad3510, 0) = -1 ENOTDIR (Not a directory)
   767  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.abi3.so", 0x7fc2ad3510, 0) = -1 ENOTDIR (Not a directory)
   768  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.so", 0x7fc2ad3510, 0) = -1 ENOTDIR (Not a directory)

   788  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/grp.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=14784, ...}, 0) = 0
   789  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/grp.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3
   810  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_struct.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=38352, ...}, 0) = 0
   811  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_struct.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3

   838  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad43a0, 0) = -1 ENOTDIR (Not a directory)
   839  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.abi3.so", 0x7fc2ad43a0, 0) = -1 ENOTDIR (Not a directory)
   840  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.so", 0x7fc2ad43a0, 0) = -1 ENOTDIR (Not a directory)

   918  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/math.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=45240, ...}, 0) = 0
   919  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/math.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3
   930  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_datetime.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=75320, ...}, 0) = 0
   931  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_datetime.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3
minghua
  • 5,981
  • 6
  • 45
  • 71

1 Answers1

0

Not sure exactly, it might be a bug in buildroot that does not resolve dependencies for this scenario. A deeper tracking of the source code seems to show: The xml and zlib packages are enabled when in configuration the python-pip is selected. However, I guess, the module info is compiled by the host-python into .pyc files and included into the target python installation. Buildroot does not rebuild host-python. An implication of that is the module info missing from the target python.

Based on the reasoning, this solution looks to work for me:

  • After enabling python-pip in configuration, run ls output/build/ | grep python to find all the python related packages.
  • On each of those packages, run a make *-dirclean on them. For example, if only python3-3.8.6 and host-python3-3.8.6 were found, run the commands below:
make host-python3-dirclean
make python3-dirclean
  • Run make again to rebuild.

No need to modify Module/Setup or Module/Setup.local.

minghua
  • 5,981
  • 6
  • 45
  • 71