5

Full Disclaimer: I have been using buildroot for the last 6 weeks. This is my first introduction to embedded Linux, thus I am still very green but have been able to solve 99% of my problems myself. for the most part the process has been straightforward.

So to occupy myself while I am stuck and home and cant work I have been working on an embedded hardware project. I've selected my hardware, built a prototype, learned buildroot basics, brought up the basic system, optimized the kernel config, built a custom device tree for my hardware and I am happy up until this point.

In parallel I have been programming the target application in python on my desktop, its dependent on a few libraries (hardware abstraction, communication, display etc) but is relatively straight forward.

I've got to the point where I have a list of requisite packages to build into my buildroot system. The buildroot tools are great here using the scanpypi tool:

~/buildroot$ utils/scanpypi diskcache -o package

Simply adding all the dependencies into /package/config.in has allowed them to be selected in menuconfig and added to the recipe.

The problem comes at build time where the building of the python module fails for the module above python-diskcache.

It has dependencies on a few things but one is slqite3, this has been added as: the core python module "sqlite module" external package "python-pysqlite3" libraries > database > sqlite

However, it fails at build:

>>> python-diskcache 4.1.0 Building
Traceback (most recent call last):
File "setup.py", line 5, in <module>
import diskcache
File "/home/buildroot/output/build/python-diskcache-4.1.0/diskcache/__init__.py", line 9, in <module>
from .core import Cache, Disk, EmptyDirWarning, JSONDisk, UnknownFileWarning, Timeout
File "/home/buildroot/output/build/python-diskcache-4.1.0/diskcache/core.py", line 14, in <module>
import sqlite3
File "/home/buildroot/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/python3.8/sqlite3/__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "/home/buildroot/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/python3.8/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
make[1]: *** [package/pkg-generic.mk:269: /home/buildroot/output/build/python-diskcache-4.1.0/.stamp_built] Error 1
make: *** [Makefile:84: _all] Error 2

In looking for a solution it seems like _sqlite3 is the C module external to python for communicating with an sqlite database. It should be installed with python (using 3.8) and should be enabled in buildroot with the enabling of the core sqlite module.

discussion 1 discussion 2

There are several fixes for dealing with this issue on a host (e.g. apt get install libsqlite3-dev and reinstall/reconfigure python). Obviously this is not possible in the image, and both the sqlite and python3 installs are latest builds and installed to the image at build time.

I'm really struggling to understand the problem or how I might patch it. I have a few theories based on the discussion, but I am unsure.

1) python is getting installed to the image before sqlite, so the appropriate module is not getting cp or symlink'ed to the python install.

2) there is some other unknown dependency not being met at build time and its failing silently

any ideas or assistance would be hugely appreciated.

Thanks

kjn
  • 51
  • 1

2 Answers2

0

The problem is that Python on your build machine finds the cross-compiled _sqlite3 module, which it can't load because it's for the wrong architecture.

This usually doesn't occur, because the setup script normally doesn't go and load the package it is trying to build/install.

One workaround could be to install the host version of all the dependencies of diskcache, and set PYTHONPATH=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/:$(PYTHON3_PATH) in DISKCACHE_ENV. However, this is liable to lead to all kinds of other breakage.

A better solution, therefore, is to patch the setup.py script of diskcache so it doesn't try to import diskcache itself. It probably only does that to get a version number or something similar; that can be solved by moving the version number into a separate file and load that one instead.

Arnout
  • 2,927
  • 16
  • 24
  • Good point, and yes, diskcache seems to import itself on setup.py. https://github.com/grantjenks/python-diskcache/blob/v5.4.0/setup.py#L27 – Jaakko Jul 03 '22 at 20:21
0

As suggested by Arnout, I got my diskcache building by creating a following patch

diff --git a/setup.py b/setup.py
index b49d9c8..22cd3c8 100644
--- a/setup.py
+++ b/setup.py
@@ -3,8 +3,6 @@ from io import open
 from setuptools import setup
 from setuptools.command.test import test as TestCommand
 
-import diskcache
-
 
 class Tox(TestCommand):
     def finalize_options(self):
@@ -23,8 +21,8 @@ with open('README.rst', encoding='utf-8') as reader:
     readme = reader.read()
 
 setup(
-    name=diskcache.__title__,
-    version=diskcache.__version__,
+    name='diskcache',
+    version='5.4.0',
     description='Disk Cache -- Disk and file backed persistent cache.',
     long_description=readme,
     author='Grant Jenks',
-- 
2.25.1
Jaakko
  • 4,674
  • 2
  • 26
  • 20