Make PyPy in virtualenv use shared library from OS rather than its own copy (libsqlite3)

Question

In a Python/Django based open source project I am working on we use Travis and GH Actions for CI. We support PyPy and hence run our CI tests for PyPy as well. Since a couple of months we were not able to run those PyPy tests successfully anymore, because we kept hitting this error: OSError: Cannot load library libgdal.so.20: /usr/lib/libgdal.so.20: undefined symbol: sqlite3_column_table_name, occuring if we would run Django's manage.py test command (traceback at end of post). Some GIS-related features in Django required the GDAL library, which in turns requrires the SQLite3 library. And it seems to requires sqlite3 to be compiled with column meta data enabled.

This was reproducible locally only if the installed sqlite3 library would be compiled without SQLITE_ENABLE_COLUMN_METADATA. After searching all installed sqlite3 libraries on the CI servers, it became clear that PyPy has its own copy of libsqlite3.so.0 installed, which it clearly prefers over the OS installed version during runtime, even though ldd would refer to the OS library:

$ ldd -d /usr/lib/libgdal.so | grep sqlite3
libsqlite3.so.0 => /usr/local/lib/libsqlite3.so.0 (0x00007f09d6124000)

I suspect the reason PyPy uses a different library (its own copy) during runtime is its dynamic loader (referenced in the traceback) and where it looks. And I suspect its own copy of libsqlite3.so.0 is created when PyPy is installed by the CI provider.

The only way to work around this that I found is by explicitly preloading the OS level library before I run the PyPy tests:

export LD_PRELOAD=/usr/local/lib/libsqlite3.so.0

This however feels like a hack and I am wondering if there is a better way to do this? Can I make PyPy (or virtualenv) just use OS libraries instead of own copies, or at least have it update its copies?

Traceback for undefined symbol error:

$ coverage run manage.py test catmaid.tests
Traceback (most recent call last):
  File "manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/core/management/__init__.py", line 377, in execute
    django.setup()
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/opt/python/pypy3.6-7.3.1/lib-python/3/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
  File "<frozen importlib._bootstrap>", line 980, in _find_and_load
  File "<frozen importlib._bootstrap>", line 964, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<builtin>/frozen importlib._bootstrap_external", line 691, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/auth/base_user.py", line 47, in <module>
    class AbstractBaseUser(models.Model):
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/base.py", line 121, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/base.py", line 325, in add_to_class
    value.contribute_to_class(cls, name)
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/options.py", line 208, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/utils.py", line 207, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/opt/python/pypy3.6-7.3.1/lib-python/3/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
  File "<frozen importlib._bootstrap>", line 980, in _find_and_load
  File "<frozen importlib._bootstrap>", line 964, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
  File "<builtin>/frozen importlib._bootstrap_external", line 691, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/travis/build/[secure]/CATMAID/django/lib/custom_postgresql_psycopg2/base.py", line 33, in <module>
    from django.contrib.gis.db.backends.postgis.base import DatabaseWrapper as PostGISDatabaseWrapper
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 6, in <module>
    from .features import DatabaseFeatures
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
    from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/base/features.py", line 3, in <module>
    from django.contrib.gis.db.models import aggregates
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/__init__.py", line 3, in <module>
    import django.contrib.gis.db.models.functions  # NOQA
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/functions.py", line 3, in <module>
    from django.contrib.gis.db.models.fields import BaseSpatialField, GeometryField
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/fields.py", line 3, in <module>
    from django.contrib.gis import forms, gdal
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module>
    from .fields import (  # NOQA
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/forms/fields.py", line 2, in <module>
    from django.contrib.gis.gdal import GDALException
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
    from django.contrib.gis.gdal.datasource import DataSource
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/datasource.py", line 39, in <module>
    from django.contrib.gis.gdal.driver import Driver
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
    from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
    from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
  File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/libgdal.py", line 46, in <module>
    lgdal = CDLL(lib_path)
  File "/opt/python/pypy3.6-7.3.1/lib-python/3/ctypes/__init__.py", line 350, in __init__
    pypy_dll = _ffi.CDLL(name, mode)
OSError: Cannot load library libgdal.so.20: /usr/lib/libgdal.so.20: undefined symbol: sqlite3_column_table_name

Django started requiring SQLite version 3.8.3 or greater in Django version 2.2. From within the PyPy CLI, try `import sqlite3; sqlite3.sqlite_version` and look at the version number? I wonder if this is related. I got a similar error when trying to run Django 2.2 with SQLite on CentOS 7, which had only had SQLite 3.7.17 underlying. — FlipperPA, Dec 28 '20 at 12:28
Thanks, I know. The problem is that even if I compile and package the latest SQLite version (with column meta data enabled) and install it in the CI OS, PyPy will still use the copy it came with (as provided by the CI service). I can only get to use the OS version with LD_PRELOAD like explained in my question. — tomka, Dec 28 '20 at 14:40
I got around this by putting the built `_sqlite.so` binary at the root of the project, so Python imports it instead of the packaged version. See: https://pypi.org/project/django-s3-sqlite/ Not sure it’ll work, but figured I’d mention it. — FlipperPA, Dec 30 '20 at 05:02
Thanks, but I went with the (in my case) simpler approach of deleting PyPy's copies of `libsqlite3.so`. — tomka, Jan 01 '21 at 19:47

score 3 · Accepted Answer · answered Dec 28 '20 at 15:32

3

PyPy is not linked with libsqlite3, but it contains the pure-Python module lib_pypy/_sqlite3.py which is based on CFFI. Try to remove (or rename) the version of libsqlite3.so that comes with PyPy; that's possibly enough. If it is not, then read on.

The pure Python logic from lib_pypy/_sqlite3.py imports _sqlite3_cffi.pypy-??.so, which is generated by CFFI by executing _sqlite3_build.py. (CFFI is also included with PyPy.) So re-running pypy _sqlite3_build.py as root should make it regenerate _sqlite3_cffi.*.so, compiled on your machine using the system-provided libsqlite3.so and sqlite3.h.

answered Dec 28 '20 at 15:32

Armin Rigo

12,048
37
48

1

Thank you very much, simply deleting all PyPy copies of `libsqlite3.so` does indeed fix the issue and is more robust than the the `LD_PRELOAD` approach, because it doesn't depend on the execution context. I now basically [call a script](https://github.com/catmaid/CATMAID/commit/0b82ffc5afa17380969da8491da055619e5a01e6) when setting up CI tests that runs the following: `sudo time find / -path /proc -prune -o -regex '.*[Pp]y[Pp]y.*libsqlite3\.so.*' -type f -exec rm -f {} +`. – tomka Jan 01 '21 at 19:42

Make PyPy in virtualenv use shared library from OS rather than its own copy (libsqlite3)

1 Answers1