1

I am currently trying to group a number of related packages together in a parent package (meta_package). While doing so I also want one of the packages to be installable as a standalone package. To do this I created the following folder structure:

├── meta_package
│   ├── subpackage1
│   │   ├── module.py
│   │   ├── __init__.py
│   ├── subpackage2
│   │   ├── module.py
│   │   └── __init__.py
│   ├── subpackage3
│   │   ├── module.py
│   │   └── __init__.py
│   └── installable_subpackage
│       ├── README.md
│       ├── __init__.py
│       ├── requirements.txt
│       ├── setup.py
│       ├── installable_subpackage
│       │   ├── __init__.py
│       │   └── submodule
│       │       ├── __init__.py
│       │       └── module.py

Although the structure above achieves the desired result, both when defining the sub-packages as namespace packages or normal packages, it introduces an extra installable_subpackage directory. As a result to import a Class from the installable_subpackage I have to use the following import statement:

from meta_package.installable_subpackage.installable_subpackage.submodule.module import Class

I, however, would like to be able to import the Class using the following (shorter) import statement:

from meta_package.installable_subpackage.submodule.module import Class

What I already tried

Use name_space packages

I tried using namespace packages for the sub-packages instead of using normal packages. This, however, did not solve the extra folder problem and also introduced a number of python import traps.

Import the redundant folder (module) inside the installable_subpackage.init.py file

I also tried importing the installable_subpackage submodule inside the installable_subpackage.__init__.py file:

import meta_package.installable_subpackage.installable_subpackage

This however does not seem to work as the meta_package.installable_subpackage.submodule import path does not point to the meta_package.installable_subpackage.installable_subpackage.submodule module. I think this is because this method only works with Classes and not modules.

Use the setuptools packages and package_dir arguments

Lastly, according to this issue, I tried to use the setuptools packages and package_dir arguments in the meta_package setup.py get rid of the extra folder. To do this I used the following setup.py:

from setuptools import setup, find_namespace_packages

setup(
    name="meta_package",
    ...
    packages=find_namespace_packages(include=["meta_package.*"]),
    package_dir={"meta_package.installable_subpackage": "meta_package/installable_subpackage/installable_subpackage"},
)

This, however, does also not seem to get rid of the extra folder.

Question

Is the packaging structure I am trying to achieve possible? Further, if so, is using it encouraged or advised against?

System information

  • Python version: Python 3.8.5
  • Virtual environment: Conda
rickstaa
  • 405
  • 5
  • 23
  • 1
    When using `package_dir`, you can"t really use `find_namespace_packages`, you have to either write the list manually or modify it before assigning it to `packages`. -- Something like this maybe: https://stackoverflow.com/a/63141103/11138259 – sinoroc Aug 18 '20 at 08:26
  • @sinoroc Thanks a lot that solved my problem! It was indeed a misunderstanding about how the `find_namespace_packages` method and the `package_dir` argument work. Do you maybe also know if in my use case it is better to use a `namespace` package or a standard package? As in my current understanding namespace packages are used primarily for distributing related packages in different folders while preventing name clashes (https://stackoverflow.com/questions/21819649/namespace-vs-regular-package). – rickstaa Aug 18 '20 at 09:51
  • @sinoroc After reading https://python.developreference.com/article/25478339/The+way+to+make+namespace+packages+in+Python I now think using a namespace package makes sense. I would like to give users the opportunity to only install the small submodule without having to download the whole library. – rickstaa Aug 18 '20 at 10:13
  • 1
    Thanks for posting all this. Clarified quite a few things for me. BTW, I was able to get this working without namespace packages, I imported the redundant subpackage using this: from .installablesubpackage.installablesubpackagemodule import * – Maniac Mar 01 '23 at 19:28

1 Answers1

2

As pointed out by @sinoroc:

When using package_dir, you can"t really use find_namespace_packages, you have to either write the list manually or modify it before assigning it to packages.

As a result to achieve the desired behaviour I had to modify the packages list before supplying it to the setuptools.setup method. This could be done in several ways.

1. Add a virtual shortened package to the packages list

We can add extra (shortened) module entries for each of the redundant folders to the packages list. This can be done by using the following setup.py file:

setup.py file

from setuptools import setup, find_namespace_packages

# Retrieve package list
PACKAGES = find_namespace_packages(include=["meta_package*"])+["meta_package.installable_subpackage"]

setup(
    name="meta_package",
    ...
    packages=PACKAGES,
    package_dir={
        "meta_package.installable_subpackage": "meta_package/installable_subpackage/installable_subpackage",
    },
)

To do this automatically for a number of sub-packages, you can use the following code:

# Retrieve package list
PACKAGES = find_namespace_packages(include=["meta_package*"])

# Add extra virtual shortened package for each of namespace_pkgs that contain redundant folders
namespace_pkgs = ["installable_subpackage"]
exclusions = r"|".join(
    [r"\." + item + r"\.(?=" + item + r".)" for item in namespace_pkgs]
)
PACKAGE_DIR = {}
for package in PACKAGES:
    sub_tmp = re.sub(exclusions, ".", package)
    if sub_tmp is not package:
        PACKAGE_DIR[sub_tmp] = package.replace(".", "/")
PACKAGES.extend(PACKAGE_DIR.keys())

setup(
    name="meta_package",
    ...
    packages=PACKAGES,
    package_dir=PACKAGE_DIR,
)

2. Modify package list such that it only contains the shortened package

Alternatively, if you want to replace the long module name with a shorter name fully, you can use the following setup.py:

setup.py file

from setuptools import setup, find_namespace_packages

# Retrieve package list
PACKAGES = find_namespace_packages(include=["meta_package*"])

# Remove redundant folders from the package list
PACKAGES = [re.sub(r"\.installable_subpackage\.(?=installable_subpackage.)", ".", package) for package in PACKAGES]

setup(
    name="meta_package",
    ...
    packages=PACKAGES,
    package_dir={
        "meta_package.installable_subpackage": "meta_package/installable_subpackage/installable_subpackage",
    },
)

This can also be done automatically for a number of sub-packages using the following code:

# Remove redundant folder from package list
red_folders = ["installable_subpackage"]
exclusions = r"|".join(
    [r"\." + item + r"\.(?=" + item + r".)" for item in red_folders]
)
PACKAGE_DIR = {}
for index, package in enumerate(PACKAGES):
    sub_tmp = re.sub(exclusions, ".", package)
    if sub_tmp is not package:
        PACKAGES[index] = sub_tmp

IMPORTANT REMARKS

  • While using the methods described above, it is important to note that they do not yet work if a package is installed in development mode (see this issue). It is, therefore, better to use the first method since developers can still use the long module name while users can also use the shorter module name.
  • Please remember that the methods described above don't work if there is an __init__.py file in your namespace package root folder (see the setuputils documentation).

Update

After I got this answer, I tried to put this logic into a setup.cfg file so that it would be PEP517/518 compatible. While doing this, I ran into some problems. The solution to this can be found on this issue I created on the setuptools GitHub page. An example repository can be found here.

rickstaa
  • 405
  • 5
  • 23