Unable to build python3-wsgiref in yocto 3.1.10

Question

I'm trying to build a Yocto image including a python service (radicale), that requires wsgiref from Python3. Wsgiref is part of Python3, however, in poky, there is no default recipe providing python3-wsgiref. In order to build it, the following Change has to be made to poky/meta/recipes-devtools/python/python3/python3-manifest.json:

-- a/python3-manifest.json
+++ b/python3-manifest.json
@@ -812,6 +812,7 @@
             "unittest",
             "unixadmin",
             "venv",
+            "wsgiref",
             "xml",
             "xmlrpc"
         ],
@@ -1196,6 +1197,37 @@
         ],
         "cached": []
     },
+    "wsgiref": {
+        "summary": "Web Server Gateway Interface Reference Implementation",
+        "rdepends": [
+            "core",
+            "crypt",
+            "datetime",
+            "email",
+            "html",
+            "io",
+            "math",
+            "mime",
+            "netclient",
+            "netserver",
+            "stringold"
+        ],
+        "files": [
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/handlers.py",
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/headers.py",
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/simple_server.py",
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/util.py",
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/validate.py"
+        ],
+        "cached": [
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/__pycache__",
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/__pycache__/handlers.*.pyc",
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/__pycache__/headers.*.pyc",
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/__pycache__/simple_server.*.pyc",
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/__pycache__/util.*.pyc",
+            "${libdir}/python${PYTHON_MAJMIN}/wsgiref/__pycache__/validate.*.pyc"
+        ]
+    },
     "xml": {
         "summary": "Python basic XML support",
         "rdepends": [

With this change, it will be provided as part of the python3 recipe.

And this is the recipe for radicale:

SUMMARY = "CalDAV and CardDAV Server"
HOMEPAGE = "https://radicale.org/"
AUTHOR = "Guillaume Ayoub <guillaume.ayoub@kozea.fr>"
LICENSE = "GPLv3"
LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"

SRC_URI = "https://files.pythonhosted.org/packages/20/d2/0761fe39ac049b1e39790b051302008eafdc7b399a39268aa2204a0b9613/Radicale-3.0.6.tar.gz"
SRC_URI[md5sum] = "fa0879b70bdcd26477759eee347ea91f"
SRC_URI[sha256sum] = "a9433d3df97135d9c02cec8dde4199444daf1b73ad161ded398d67b8e629fdc6"

S = "${WORKDIR}/Radicale-3.0.6"

RDEPENDS_${PN} = "python3-defusedxml python3-passlib python3-vobject python3-python-dateutil python3-setuptools python3-fcntl python3-wsgiref"

inherit setuptools3

As you can see, python3-wsgiref is appended to RDEPENDS_${PN} and when I build the whole image, everything builds fine and the service is able to start on the system.

However, I would like to put the change to python3-manifest.json into a patch and apply the patch in a bbappend file. For this I added a bbappend in my custom "meta-radicale" layer, 'meta-radicale/recipes-devtools/python/python3_3.8.11.bbappend':

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
SRC_URI_append = " file://build-wsgiref.patch;patchdir=${WORKDIR} "

The patch shown above is put into 'meta-radicale/recipes-devtools/python/python3/build-wsgiref.patch'.

This is the whole file tree of meta-radicale:

meta-radicale/
meta-radicale/recipes-radicale
meta-radicale/recipes-radicale/python
meta-radicale/recipes-radicale/python/python3-versions.inc
meta-radicale/recipes-radicale/python/python3-passlib_1.7.4.bb
meta-radicale/recipes-radicale/python/python3-six_1.16.0.bb
meta-radicale/recipes-radicale/python/python3-vobject_0.9.6.1.bb
meta-radicale/recipes-radicale/python/python3-python-dateutil_2.8.2.bb
meta-radicale/recipes-radicale/python/python3-radicale_3.0.6.bb
meta-radicale/recipes-radicale/python/python3-defusedxml_0.7.1.bb
meta-radicale/recipes-devtools
meta-radicale/recipes-devtools/python
meta-radicale/recipes-devtools/python/python3
meta-radicale/recipes-devtools/python/python3/build-wsgiref.patch
meta-radicale/recipes-devtools/python/python3_3.8.11.bbappend
meta-radicale/conf
meta-radicale/conf/layer.conf

The recipes under 'meta-radicale/recipes-radicale/python/' have been created by 'pipoe -p radicale --python python3' and work fine, I tested them.

If I try to build the image, I get the following output:

RROR: smokylinux-image-1.0-r0 do_rootfs: Could not invoke dnf. Command '/home/imanoid/workspace/smoky-pi/build/smoky-qemuarm64/tmp/work/qemuarm64-poky-linux/smokylinux-image/1.0-r0/recipe-sysroot-native/usr/bin/dnf -v --rpmverbosity=info -y -c /home/imanoid/workspace/smoky-pi/build/smoky-qemuarm64/tmp/work/qemuarm64-poky-linux/smokylinux-image/1.0-r0/rootfs/etc/dnf/dnf.conf --setopt=reposdir=/home/imanoid/workspace/smoky-pi/build/smoky-qemuarm64/tmp/work/qemuarm64-poky-linux/smokylinux-image/1.0-r0/rootfs/etc/yum.repos.d --installroot=/home/imanoid/workspace/smoky-pi/build/smoky-qemuarm64/tmp/work/qemuarm64-poky-linux/smokylinux-image/1.0-r0/rootfs --setopt=logdir=/home/imanoid/workspace/smoky-pi/build/smoky-qemuarm64/tmp/work/qemuarm64-poky-linux/smokylinux-image/1.0-r0/temp --repofrompath=oe-repo,/home/imanoid/workspace/smoky-pi/build/smoky-qemuarm64/tmp/work/qemuarm64-poky-linux/smokylinux-image/1.0-r0/oe-rootfs-repo --nogpgcheck install gerbera imanoid openssh openssh-sftp-server packagegroup-core-boot python3-radicale run-postinsts sudo vim' returned 1:
DNF version: 4.2.2
cachedir: /home/imanoid/workspace/smoky-pi/build/smoky-qemuarm64/tmp/work/qemuarm64-poky-linux/smokylinux-image/1.0-r0/rootfs/var/cache/dnf
Added oe-repo repo from /home/imanoid/workspace/smoky-pi/build/smoky-qemuarm64/tmp/work/qemuarm64-poky-linux/smokylinux-image/1.0-r0/oe-rootfs-repo
repo: using cache for: oe-repo
not found other for: 
not found modules for: 
not found deltainfo for: 
not found updateinfo for: 
oe-repo: using metadata from Mon 23 Aug 2021 01:56:26 PM UTC.
No module defaults found
--> Starting dependency resolution
--> Finished dependency resolution
Error: 
 Problem: conflicting requests
  - nothing provides python3-wsgiref needed by python3-radicale-3.0.6-r0.aarch64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

ERROR: Logfile of failure stored in: /home/imanoid/workspace/smoky-pi/build/smoky-qemuarm64/tmp/work/qemuarm64-poky-linux/smokylinux-image/1.0-r0/temp/log.do_rootfs.4376
ERROR: Task (/home/imanoid/workspace/smoky-pi/yocto/meta-smokylinux/recipes-smokylinux/images/smokylinux-image.bb:do_rootfs) failed with exit code '1'

I don't understand why the behaviour is different, when modifying python3-manifest.json directly vs modifying it via patch.

Answer 1

After some more googling, I think I found an answer to my problem: https://hub.mender.io/t/how-to-work-with-python-applications-and-modules-in-yocto-project/1135

"You can get a list of these individual packages by reading this file:

cat ../sources/poky/meta/recipes-devtools/python/python3/python3-manifest.json

If the standard library module is not listed in the above file, it is part of the python3-misc package."

Apparently, the default python3 recipe puts all the standard python modules that aren't part of the manifest automatically into a package called 'python3-misc'. So the only thing I need to do is add 'python3-misc' instead of 'python3-wsgiref' to the RDEPENDS line of python3-radicale_3.0.6.bb:

RDEPENDS_${PN} = "python3-defusedxml python3-passlib python3-vobject python3-python-dateutil python3-setuptools python3-fcntl python3-misc"

Now it compiles successfully and the wsgiref module can be imported in the target system.

However, as to why there is a difference between patching and editing the manifest, I haven't found out.