How to structure Python library and path in the same way as MATLAB

Question

Over the years I have written a few hundred functions in matlab for space engineering purposes which I use on a daily basis. They are all nicely put in one folder ordered in subfolders, and in matlab I just have an addpath() command for the root folder in the startup.m file, and then I can use any of the functions right away after starting up matlab.

I am now trying to do the same in python.

As far as I understand in python I shouldn't have 1 file for every function like in matlab, but rather bundle all the functions together in 1 py file. Is this correct? I am trying to avoid this, since I have a strong preference for short scripts rather than 1 huge one, due to it being way more intuitive for me that way.

And then, once I have all my python scripts, can I place them anywhere in order to use them? Because I read that python works differently than matlab in this aspect, and scripts need to be in the working directory in order to import them. However, I want to load the scripts and be able to use them regardless of my active working directory. So I am guessing I have to do something with paths. I have found that I can append to pythonpath using sys.path.insert or append, however this feels like a workaround to me, or is it the way to go?

So considering I have put all my rewritten matlab fuctions in a single python file (lets call it agfunctions.py) saved in a directory (lets call it PythonFunctions). The core of my startup.py would then be something like (I have added the startup file to PYTHONSTARTUP path):

# startup.py
import os, sys
import numpy as np
import spiceypy as spice

sys.path.append('C:\Users\AG5\Documents\PythonFunctions')
import agfunctions as ag

Does any of this make sense? Is this the way to go, or is there a better way in python?

Answer 1

Well, the python package is probably the best way to solve your problem. You can read more here. Python packages have no need to be built and not always are created for sharing, so do not worry.

Assume you had this file structure:

Documents/
  startup.py
  PythonFunctions/
    FirstFunc.py
    SecondFunc.py

Then you can add file __init__.py in your PythonFunctions directory with next content:

__all__ = ['FirstFunc', 'SecondFunc']

Init file must be updated if you change filenames, so maybe it isn't best solution for you. Now, the directory looks like:

Documents/
  startup.py
  PythonFunctions/
    __init__.py
    FirstFunc.py
    SecondFunc.py

And it's all - PythonFunctions now is a package. You can import all files by one import statement and use them:

from PythonFunctions import *

result_one = FirstFunc.function_name(some_data)
result_two = SecondFunc.function_name(some_other_data)

And if your startup.py is somewhere else, you can update path before importing as next:

sys.path.append('C:\Users\AG5\Documents')

More detailed explanations can be found here

Answer 2

There is no need to write all your matlab functions within one file. You can also keep (kind of) your desired folder structure of your library. However, you should write all functions in the deepest subfolder into one *.py file.

Suppose your MATLAB library is in the folder space_engineering and is set up like this:

space_engineering\
    subfolder1\
        functiongroup1\
            printfoo.m
            printbar.m
    subfolder2\
        ...
    subfolder3\
        ...
    ...

After doing addpath(genpath('\your_path\space_engineering')) all functions in subfolders subfolder* and functiongroup* are available in the global namespace like this

>> printfoo    % function printfoo does only do fprintf('foo')
foo

I understand this is an behaviour your want to preserve in your migrated python library. And there is a way to do so. Your new python library would be structured like this:

space_engineering\
    __init__.py
    subfolder1\
        __init__.py
        functiongroup1.py
        functiongroup2.py
    subfolder2\
        __init__.py
        ...
    subfolder3\
        __init__.py
        ...
    ...

As you can see the deepest subfolder layers functiongroup* of the MATLAB structure is replaced now by functiongroup*.py files, so called modules. The only compromise you have to allow for is that functions printfoo() and printbar() are now defined in this .py modules instead of having individual .py files.

# content of functiongroup1.py
def printfoo():
    print(foo)

def printbar():
    print(bar)

To allow for doing the same function calling as in MATLAB you have to make the function names printfoo and printbar available in the global namespace by adjusting the __init__.py files of each subfolder

# content of space_enginieering\subfolder1\__init__.py
from .functiongroup1 import *
from .functiongroup2 import *

as well as the __init__.py of the main folder

# content of space_engineering\__init__.py
from .subfolder1 import *
from .subfolder2 import *
from .subfolder3 import *

The from .functiongroup1 import * statement loads all names from module functiongroup1 into the namespace of subfolder1. Successively from .subfolder1 import * will forward them to the global namespace.

Like this you can do in an python console (or in any script) e.g.:

>>> sys.path.append('\your_path\space_engineering')
>>> from space_engineering import *
>>> printfoo()
foo

This way you can use your new python library in the same way as you former used the MATLAB library.

---

HOWEVER: The usage of from xyz import * statement is not recommend in python (see here why, it is similar to why not using eval in MATLAB) and some Pythonistas may complain recommending this. But for your special case, where you insist on creating a python library with MATLAB like comfort, it is a proper solution.