2

I'm writing a couple of packages that I'd like to release on PyPi for other people to use.

I've not released to PyPi before so I have been mocking up a submission template: https://github.com/chris-brown-nz/pypi-package-template

Here's a tree of the project template:

|   MANIFEST.in
|   README.rst
|   setup.cfg
|   setup.py
|       
\---package
        module_one.py
        module_three.py
        module_two.py
        __init__.py

In terms of interacting with the package, this is what I would usually do - is it the best way?

To run a method:

from package import module_one
module_one.ClassOne().method_a()

To get a value from a method:

from package import module_two
print(module_two.ClassFive().method_e())

To set then use an attribute of an instance:

from package import module_three
cls = module_three.ClassSeven("Hello World")
print(cls.value)

'package' is a reserved name obviously and won't be used in the final project.

I'd be grateful for some feedback on how I've structured my project and whether it is considered standard, or if it should be modified in some way.

1 Answers1

1

There are different approaches to this, whether one or the other is better is depending on a how you want to develop, usage of the package (e.g. if you ever install it using pip install -e packag_name), etc.

What is missing from your tree is the name of the directory where the setup.py resides, and that is usually the package name:

└── package
    ├── package
    │   ├── __init__.py
    │   ├── module_one.py
    │   ├── module_three.py
    │   └── module_two.py
    ├── MANIFEST.in
    ├── README.rst
    ├── setup.cfg
    └── setup.py

as you can see you are doubling the 'package' name, and that means that your setup.py has to be adapted for each package, or dynamically determine the name of the directory where the module.py files resides. If you go for this route, I would suggest you put the module.py files in a generically named directory 'src' or 'lib'.

I don't like the above "standard" setup for multiple reasons:

  • it doesn't map well to how python programs "grow" before they are split up into packages. Before splitting up having such a 'src' directory would mean using:

    from package.src.module_one import MyModuleOneClass
    

    Instead you would have your module.py files directly under package

  • Having a setup.py to control installation, a README.rst for documentation and an __init__.py to satisfy Python's import is one thing, but all other stuff, apart from your module.py files containing the actual functionality, is garbage. Garbage that might be needed at some point during the package creation process, but is not necessary for the package functionality.

There are other considerations, such as being able to access the version number of the package from the setup.py as well as from the program, without the former having to import the package itself (which may lead to install complications), nor having another extra version.py file that needs importing.

In particular I always found the transition from using a directory structure under site-packages that looked like:

└── organisation
    ├── package1
    └── package2
        ├── subpack1
        └── subpack2

and that could intuitively be used for both importing and navigation to source files, to something like:

├── organisation_package1
│   └── src
├── organisation_package2_subpack1
│   └── src
└── organisation_package2_subpack2
    └── src

unnatural. To rearrange and break a working structure to be able to package things seems wrong.

For my set of published packages I followed another way: - I kept the natural tree structure that you can use "before packaging", 'src' or 'lib' directories. - I have a generic setup.py which reads and parses (it does not import) the metadata (such as version number, package name, license information, whether to install a utility (and its name)), from a dictionary in the __init__.py file. A file you need anyway. - The setup.py is smart enough to distinguish subdirectories containing other packages from subdirectories that are part of the parent package. - setup.py generates files that are needed during package generation only (like setup.cfg), on the fly, and deletes them when no longer needed.

The above allows you to have nested namespaced packages (i.e. package2 can be a package you upload to PyPI, in addition to package2.subpack1 and package2.subpack2). The major thing it (currently) doesn't allow is using pip install -e to edit a single package (and not have the others editable). Given the way I develop, that is not a restriction.

The above embraces namespace packages, where many other approaches have problems with these (remember the last line of Zen of Python: Namespaces are one honking great idea – let’s do more of those)

Examples of the above can e.g be found in my packages ruamel.yaml (and e.g. ruamel.yaml.cmd), or generically by searching PyPI for ruamel.


As is probably obvious, the standard disclaimer: I am the author of those packages

As I use a utility to start packaging, which also runs the tests and does other sanity checks, the generic setup.py could be removed from the setup and inserted by that utility as well. But since subpackage detection is based upon setup.py availability or not, this requires some rework of the generic setup.py.

Anthon
  • 69,918
  • 32
  • 186
  • 246
  • Thanks Anthon, that's a lot to get my head around but I've had a look through your bitbucket repo and can kind of see how it's working. A question though, for my situation there will only ever be one package to install, and no subpackages. My understanding of your model is that I'd have one directory ('package') with everything beneath this (and no subfolders other than _doc and _test etc?) –  Feb 21 '17 at 20:46
  • Since you indicated you have a template, I thought you implicated multiple packages. For just one package that will go to PyPI you don't really need a template. You can still use my `setup.py` even without subpackages (or multiple packages), but it doesn't pay off that much. If you have more questions ask here, or sent me an email (you can find that via profile). – Anthon Feb 21 '17 at 21:39
  • Thanks Anthon, sorry I didn't make that clear in the first place. I do actually have three packages I'm developing but they're largely unrelated so I'll keep them separate. Do you think I'm on the right track with how I intend users to interact with the package, as per the examples in my question? –  Feb 24 '17 at 04:38