Get Python to look in different location for Lib using Py_SetPath()

Question

I have embedded Python in an application, foo.exe. When it runs, the Python is invoked and immediately looks for Lib. The only way I can get it to work is to place Lib (Python's Directory Library of modules) in the location as foo.exe.

Is there a way I can redirect Python to look somewhere else, such as Python/Lib ? I am not able to change PATH (This is windows) and I don't want to hack the Python source code.

Basically, I cannot get Py_SetPath() to work, and I have not been able to find any practical examples on the internet.

Update:

OK, this works:

  #define MYMAXPATHLEN 1000
  static wchar_t progpath[MYMAXPATHLEN + 1];

  wchar_t* pdir = L"\\My_New_Location\\Python\\Lib";
  wchar_t* pdelim = L";";
  wchar_t* pypath = NULL;
  GetModuleFileNameW(NULL, progpath, MYMAXPATHLEN);
  reduce(progpath);
  wcscat(progpath,pdir);
  // I get the present module path and add the extra dirs to access Lib code

  wcscat(progpath, pdelim);  // I add a path delimiter

  pypath = Py_GetPath();
  wcscat(progpath, pypath);
  // I add the paths that Py_GetPath() produces.

  Py_SetPath(progpath);

  Py_Initialize();

I also call Py_SetProgramName(); AFTER Py_Initialize(); I am not sure if all this extra stuff is needed, but smaller solutions seem to fail.

It seems that calling Py_SetProgamName() AFTER the initialize is very important to having the embedding call working properly.

If `Lib` doesn't exist, then maybe you could create a junction `mklink /J c:\...\Lib c:\...\Python\Lib`? — ldavid, Apr 26 '18 at 23:24
From the Python code: "Py_SetPath() can be used to override this mechanism. Call Py_SetPath with a semicolon separated path prior to calling Py_Initialize." — Jiminion, Apr 27 '18 at 13:45

score 2 · Answer 1 · edited Apr 26 '18 at 23:19

2

Before importing the library, run the following line:

sys.path.append('C:\path to Lib')

Details can be found here.

edited Apr 26 '18 at 23:19

Abe

1,357
13
31

answered Apr 26 '18 at 20:29

python_enthusiast

324
3
15

I think it has to be earlier in the process (a certain Python API call?) Python fails before a command can be sent to it. – Jiminion Apr 26 '18 at 20:44

score 1 · Answer 2 · answered Nov 19 '19 at 17:44

I got it to work (on Linux) doing the following:

// method to inspect PyObjects 
static void reprint(PyObject *obj) 
{
    PyObject* repr = PyObject_Repr(obj);
    PyObject* str = PyUnicode_AsEncodedString(repr, "utf-8", "~E~");
    const char *bytes = PyBytes_AS_STRING(str);

    printf("REPR: %s\n", bytes);
    Py_XDECREF(repr);
    Py_XDECREF(str);
}

.

int main()
{
// for our manually compiled and installed usr-local Python 3.6
//#define PATH            L"/usr/local/bin:/usr/local/lib:/usr/local/lib/python3.6/lib-dynload"
//#define PREFIX          L"/usr/local"
//#define EXEC_PREFIX     L"/usr/local"
//#define FULL_PROG_PATH  L"/usr/local/bin/python3.6"

// for apt installed Python 3.7
//#define PATH            L"/usr/bin:/usr/lib:/usr/lib/python3.7/lib-dynload"
//#define PREFIX          L"/usr"
//#define EXEC_PREFIX     L"/usr"
//#define FULL_PROG_PATH  L"/usr/bin/python3.7"

// for venv (which uses the /usr/local/lib python 3.6)
#define PATH            L"/home/me/venv_dir/bin:/home/me/venv_dir/lib:/usr/local/lib/python3.6:/usr/local/lib/python3.6/lib-dynload"
#define PREFIX          L"/home/me/venv_dir"
#define EXEC_PREFIX     L"/usr/local"
#define FULL_PROG_PATH  L"/usr/local/bin/python3.6"

// ------------------------------------------------

#define CHANGE_THE_INTERPRETER

#ifdef CHANGE_THE_INTERPRETER

// TODO : Look at using this: https://www.python.org/dev/peps/pep-0587/

Py_SetPath(PATH);

// change the built-in prefix/exec-prefix, in place.
wchar_t* wpPrefix     = Py_GetPrefix();
wchar_t* wpExecPrefix = Py_GetExecPrefix();
wchar_t* wpProgramFullPath = Py_GetProgramFullPath();
wcscpy (wpPrefix, PREFIX);
wcscpy (wpExecPrefix, EXEC_PREFIX);
wcscpy (wpProgramFullPath, FULL_PROG_PATH);

#endif //CHANGE_THE_INTERPRETER

// inspect the environment variables.  With the #define commented out above the "defaults" appear as indicated to the right hand side
wchar_t* xx;                        //defaults
xx = Py_GetPrefix();                //<prefix> L"/usr/local"
xx = Py_GetExecPrefix();            //<exec_prefix> L"/usr/local"
xx = Py_GetPath();                  //L"/usr/local/lib/python36.zip:/usr/local/lib/python3.6:/usr/local/lib/python3.6:/usr/local/lib/python3.6/lib-dynload"
xx = Py_GetProgramName();           //L"python3"
xx = Py_GetPythonHome();            //null
xx = Py_GetProgramFullPath();       //<progpath> L"/usr/local/bin/python3"

Py_Initialize();

some extra bits

int x2 = PyRun_SimpleString ("import site; print (site.getsitepackages())");
int x3 = PyRun_SimpleString ("import datetime");
int x4 = PyRun_SimpleString ("import numpy as np");

//inspect sys info
PyObject* sys_executable = PySys_GetObject((char*)"executable");  reprint(sys_executable);
PyObject* sys_version    = PySys_GetObject((char*)"version");  reprint(sys_version);

PyObject* sys_realPrefix = PySys_GetObject((char*)"real_prefix");  reprint(sys_realPrefix);
PyObject* sys_basePrefix = PySys_GetObject((char*)"base_prefix");  reprint(sys_basePrefix);

Some points to note:

If you look at the Python module getpath.c you will see the following buffers:

static wchar_t prefix[MAXPATHLEN+1];
static wchar_t exec_prefix[MAXPATHLEN+1];
static wchar_t progpath[MAXPATHLEN+1];

Methods such as Py_GetProgramFullPath perform as follows:

if (!module_search_path)
    calculate_path();
return progpath;

...so it is possible to use those methods to obtain the buffer pointers and wcscpy the values directly into the buffers. Note this is currently only possible with getpath being implemented in this fashion!

lib-dynload is needed to ensure that some modules (e.g. datetime) can be pulled in

Also note this approach ensures that the .../python3.x/encodings directory can be found, which prevents a runtime error within Py_Initialize

Get Python to look in different location for Lib using Py_SetPath()

2 Answers2

Some points to note: