Python's Namespace Packages are not Visible in Local Development Project

Question

I'm trying to split a big (almost monolithic) Python project into multiple pieces by utilizing Pythons namespace packages. Therefore I extracted a parser for *.rules files into a namespace package. Python calls this a distribution.

I followed that guide and as far as I can tell it partially works, but ...

In short: In the main project, the namespaces of separately distributed namespace packages are not visible, because the package search finds local packages first and does not merge with the system packages from site-packages.

Project Structure

Here is a part of my directory / package and module structure.

Main project:

pyIPCMI/                  # Git repository root
  pyIPCMI/
    __init__.py
    Common/
      __init__.py
      File1.py
    Compiler/
      __init__.py
      Vendor1.py
      Vendor2.py
  setup.py

Distribution for rules parser

pyIPCMI.Parser.Rules/     # Git repository root
  pyIPCMI/
    Parser/
      Rules/
        __init__.py
        Parser.py
  setup.py

Package Descriptions (setuptools)

The main project is packaged like this:

import setuptools

setuptools.setup(
  name="pyIPCMI",
  version="1.1.5",
  author="Paebbels",
  author_email="abc@xyz.de",
  description="",
  long_description="",
  url="https://github.com/Paebbels/pyIPCMI",

  packages=setuptools.find_packages(),

  classifiers=["License :: OSI Approved :: Apache Software License"],
  python_requires='>=3.5',
  install_requires=[],
)

The embedded namespace is packaged like this:

import setuptools

namespace =   ["pyIPCMI", "Parser", "Rules"]

setuptools.setup(
  name=".".join(namespace),
  version="1.1.4",
  author="Paebbels",
  author_email="abc@xyz.de",
  description="",
  long_description="",
  url="https://github.com/Paebbels/pyIPCMI.Parser.Rules",

  packages=setuptools.find_namespace_packages(
    include=[".".join(namespace), ".".join(namespace) + ".*"]
  ),
  namespace_packages=namespace[0:1],

  classifiers=["License :: OSI Approved :: Apache Software License"],
  python_requires='>=3.5',
  install_requires=[],
)

All distributions have been:

• packaged with setuptools using Travis-CI
• deployed to PyPI, and then
• installed on the local machine using pip.

Namespace Structure at PyPI

pyIPCMI
pyIPCMI.Parser.Files
pyIPCMI.Parser.Rules
pyIPCMI.Toolchains
pyIPCMI.Toolchains.Vendor1
pyIPCMI.Toolchains.Vendor2

Problem Description

When developing in the main project with e.g. PyCharm, the locally found namespaces are preferred over packages from site-packages. Moreover, these namespace don't get merged. Because the main project has the same root namespace pyIPCMI, search continues in the local development project but does not search in site-packages.

Any idea how to be able to develop the main project?

---

Please advise what information is needed to solve this question. I tried to write down all information I have so far. But this question may need improvement to get a solution.

Answer 1

I ran into the same problem last week and found the solution. As first example, let me give you my initial (non-working) layout.

Directory Structure 1 (plugins not yet extracted)

project-dir
  +-- pyproject.toml/setup.py
  +-- myproject
        +-- __init__.py
        +-- app.py
        +-- plugins
              +-- __init__.py
              +-- plugin1
                    +-- __init__.py
                    +-- plugmodule.py
              +-- plugin2
                    +-- __init__.py
                    +-- plugmodule.py

The application already had code to find plugin modules based on this structure. When writing this, my initial understanding was that I could provide additional plugins by creating Python-packages with the following structure:

plugin-package-dir
  +-- pyproject.toml/setup.py
  +-- myproject
        +-- plugins
              +-- plugin3
                    +-- __init__.py
                    +-- plugmodule.py

As mentioned in the official docs, the namespace packages don't contain __init__.py files here.

But this did not work

Solution

In order to make this work, I needed to extract the "plugins" package completely from the main project subtree by making this a pure namespace packages as well. This new package then needs to be manually added to your setup.py/pyproject.toml as it is not recognized by default package discovery.

I ended up with the following structure:

project-dir
  +-- pyproject.toml/setup.py
  +-- myproject
        +-- __init__.py
        +-- app.py
  +-- myproject_plugins
        +-- plugins
              +-- plugin1
                    +-- __init__.py  # <-- not strictly needed? (see note)
                    +-- plugmodule.py
              +-- plugin2
                    +-- __init__.py  # <-- not strictly needed? (see note)
                    +-- plugmodule.py

And then for the plugin packages I have this:

plugin-package-dir
  +-- pyproject.toml/setup.py
  +-- myproject_plugins
        +-- plugins
              +-- plugin3
                    +-- plugmodule.py
              +-- plugin4
                    +-- plugmodule.py

NOTE: The __init__.py files exist in the main project plugins because of the way I load plugins. In order to iterate over the namespace I import it during plugin discovery. I could instead catch the ImportError and report that no plugin was found at all.

I have not investigated 100% if I can remove the __init__.py and catch the import error. I have lost already too much time on this and don't want to break something that's working right now ;)

Answer 2

As far as I know the namespace packages have to be really empty, and this in all installed projects.

Apparently your pyIPCMI project has a top level non-namespace-package pyIPCMI, i.e. you have a pyIPCMI/__init__.py file. So I don't think you can have a pyIPCMI namespace package in other projects. I would try removing this pyIPCMI/__init__.py file (and make it a namespace package as well) if you want pyIPCMI to be a namespace package in the other projects.