9

Current situation

I embedded Python into my C++ application. The application is installed into the common Application locations on Windows, macOS (C:\Program Files, /Applications/, ...) which require admin privileges to make modifications to these directories. My application also ships with its own Python installation. Here is an example for the packaging structure:

C:\Program Files\Foo\foo.exe
C:\Program Files\Foo\Lib
C:\Program Files\Foo\DLLs
C:\Program Files\Foo\python.exe

The program also sets up an additional user-specific directory, which doesn't require any permissions and which is in sys.path:

C:\Users\username\AppData\Roaming\Foo\python\libs

Problem

During initialization I call PySet_Path, Py_SetPythonHome and I set sys.prefix to Program Files\Foo (as required by the documentation)

If the user installs pip or a module through pip it ends up in this application directory which would requires elevated permissions rights of course. But there is pip install something --user to find a user friendly installation directory (which is the AppData dir in my case).

My question is, which values and combinations should I set to make use of --user? Is it recommended to set the Python home to the AppData directory, and sys.prefix to the Program Files directory?

Because, that would mean the user could use --user to control where certain files are installed to

C:\Program Files\Foo\python.exe -m pip install xyz --user
would go into the user directory

C:\Program Files\Foo\python.exe -m pip install xyz
would go into the Program Files directory

Is this recommended, or am I overseeing any side-effects here?

Daniel Stephens
  • 2,371
  • 8
  • 34
  • 86
  • no it is not recommended. I am not sure if home can be set dynamicaly each time app runs, instead of altering permanently the user settings for default python – Nikos M. May 23 '20 at 18:02
  • 1
    In an embedded environment, the Python home is always set dynamically/manually during startup. What defines the Python home not to be in AppData? Just curious, because the Lib/ or DLLs dir has to be located there? Technically that could also be in the AppData dir – Daniel Stephens May 23 '20 at 18:28
  • 2
    Perhaps the https://pypi.org/project/xdgappdirs/ project would help by decoupling your code from the general part of the problem? – Shaheed Haque May 28 '20 at 16:03
  • Very interesting module! Bookmarked, unfortunately it doesn't solve the issue because I would need to ship everything without such external dependency + pythonhome is not covered, but again, thank you! – Daniel Stephens May 29 '20 at 01:25
  • It seems to me that you are the one that created this totally unnecessary problem. *Python* is (can be) installed in the same manner. Why not preserve the behavior, and set all those paths? Instead you could leave the "defaults" (and your app act like the interpreter itself) and thus saving you of any headaches like this one. If a anyone has a question, simply point them to the official *doc*s. Also note that "*embedded environment*" is a bit misleading ("*embedded Python*" would probably be more appropriate). – CristiFati May 31 '20 at 09:25
  • 1
    I would disagree on that one. The problem of Python ARE the defaults when embedded. I can relate to the problem, but don't have a solution to this either – HelloWorld Jun 01 '20 at 01:45
  • Use the embedded Python on Windows, turn on the user site. `pip install --user` will install packages into `AppData\Roaming\Python\PythonXY\site-packages` which is shared between different Python installations. This way, if the user has its own Python installation and installed packages for it, he can automatically reuse the packages for the embedded installation. We and our users are pretty happy with this model in our app. – hoefling Jun 01 '20 at 08:25
  • Unfortunately that doesn't work. Virtual environments don't work as well with embedded interpreters and therefore collisions between library versions are just a matter of time – Daniel Stephens Jun 01 '20 at 19:38

2 Answers2

2

I previously embedded a python interpreter into a Java application. I used Virtual Environments to manage things. This gives a lot of control over packages and where things end up.

Essentially you can have things in the system and/or override them in the virtualenv. And you can have virtualenv's for each user if needed.

See here for more about it:

https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/

user2199860
  • 788
  • 4
  • 14
  • I use virtual environment for System Python installations, but not for embedded interpreters. Do you know any other resources that could help me on to connect virtual envs and my embedded interpreter? – Daniel Stephens Jun 01 '20 at 19:40
  • 1
    You can find some pointers here: https://bugs.python.org/issue35706 – user2199860 Jun 01 '20 at 20:07
1

As you have mentioned that the embedded system ships with its own python environment, thus it is not recommended to use pip to install libraries outside the environment.

That being said,

C:\Program Files\Foo\python.exe -m pip install xyz --user

The above command shall affect the system-wide behaviour of the Python environment for the user, which may lead to unexpected behaviour of the application as well as the Operating System. Also, this installation would mean that the installed application shall behave in a certain way for each user which may or may not be the same.

However,

C:\Program Files\Foo\python.exe -m pip install xyz

using this installation method helps in delivering similar behaviour of the application among users which is subjective as, if anything goes bad for one user, the application stops performing for all the users.

Shikhar Bhardwaj
  • 11
  • 2
  • Thanks for your answer! But the purpose of a custom `sys.prefix` would be to change it to a custom path so it doesn't conflict with the system-wide Python installation. In my case it would be somewhere in `Roaming\Foo\` under Windows. – Daniel Stephens Jun 01 '20 at 19:42
  • Yes, you are absolutely correct. `sys.prefix` would serve the purpose, but this option is similar to the installation of libraries inside the python environment of the embedded system. You'll just be pointing your system to look for indexing of libraries from some other directory.. Thus the behaviour of the application for all the users shall be the same. – Shikhar Bhardwaj Jun 01 '20 at 19:58