15

I'm trying to use pyInstaller to package a wxpython application. I looking for a variation of the "one-folder" mode whereby the dlls and pyds are not stored in the top-level directory but in a subdirectory instead (like "dlls" or "libs").

This is the spec file currently:

# -*- mode: python -*-
import os

a = Analysis\
(
    ["..\\job_scraper\\load_gui.py"],
    pathex        = ["C:\\Users\\Administrator\\Documents\\Projects\\python\\PyInstaller\\load_gui"],
    hiddenimports = [],
    hookspath     = None,
    runtime_hooks = None
)

a_binaries = []
for (name, path, data_type) in a.binaries:
    (non_ext, ext) = os.path.splitext(name)
    if(ext in [".pyd", ".dll"]):
        a_binaries.append((os.path.join("libs", name), path, data_type))
    else:
        a_binaries.append((name, path, data_type))

a.binaries = TOC(a_binaries)

pyz = PYZ(a.pure)

exe = EXE\
(
    pyz,
    a.scripts,
    exclude_binaries = True,
    name             = "load_gui.exe",
    debug            = False,
    strip            = None,
    upx              = True,
    console          = False
)

coll = COLLECT\
(
    exe,
    a.binaries,
    a.zipfiles,
    a.datas,
    [("control.csv", "..\\job_scraper\\control.csv", "DATA")],
    strip = None,
    upx   = True,
    name  = "load_gui"
)

This does to put the dlls (not the pyds) into a lib folder, however it seems to do this after linking and so the program fails to launch because it can't find the expected dlls.

dilbert
  • 3,008
  • 1
  • 25
  • 34
  • I am facing the same Problem. Have you found any solution yet? – Mailerdaimon Nov 18 '13 at 13:42
  • 2
    I've looked into it. It requires modification of the bootloader source code. There is a section in the code that defines where it looks for .pyd, .dll, etc. I haven't made any progress beyond that, as I haven't been able to get MingW to build the bootloader. – dilbert Nov 18 '13 at 23:54
  • Too bad, this would be a nice option to have in pyinstaller. I'll see if I can work on that at some point. – Iguananaut Aug 29 '17 at 17:01

2 Answers2

6

The problem is that the sys.path doesn't include your subdirectories. So when the program runs, it doesn't know where to look for your .dll or .pyd files.

Of course putting the code sys.path.append("relative/path/to/your/subdirectories") on top of your main script would come to mind. But one again, this code is only executed after everything is loaded and in place.

According to this blog, the solution is using the runtime hook of pyinstaller. Runtime hook tells the bootstrapping code to run any of your arbitrary code before your main script runs - before importing any stuff.

1. Preparation

Create a hooker.py which will add all your custom paths to sys.path. Put it somewhere and do it 1 time only.

import sys
import os
sys.path.append(os.path.join(os.path.dirname(sys.argv[0]), "lib"))    # for pyd
sys.path.append(os.path.join(os.path.dirname(sys.argv[0]), "windll")) # for dll

2. Tell pyinstaller to include the hooker.py

with spec file:

a = Analysis\
(
    ["..\\job_scraper\\load_gui.py"],
    pathex        = ["C:\\Users\\Administrator\\Documents\\Projects\\python\\PyInstaller\\load_gui"],
    hiddenimports = [],
    hookspath     = None,
    runtime_hooks = "absolute/path/to/hooker.py"  # <----- add it here
)

or with commandline:

pyinstaller --runtime-hook="absolute/path/to/hooker.py" the_rest_parameters

3. Run pyinstaller

As usually, it will create dist/your_main_script_name folder which contains the exe file, manifest, library.zip, and a bunch of .dll and .pyd

4. Create custom folders

Now you can create a windll folder and lib or anything that you add to sys.path at step 1. Then move all .pyd files to lib and all .dll files to windll.

Run your exe and it will crash! So move back these below files to parent folder.

  • pythonXX.dll , where XX is your python version
  • VCRUNTIME140.dll
  • pywintypesXX.dll, where XX is your python version and if it is included

These files are needed for the bootstrap so we cannot move them without modifying the bootstrap code.

Run the exe again and it should work normally.

Soon you will get bored of repeating all of the above over and over again. So here is what I have been doing.

Create compiler.bat with the content similar to:

pyinstaller --runtime-hook="absolute/path/to/hooker.py" --onedir --icon path/to/icon ^
--exclude-module=UnNeeded_module_A ^
--exclude-module=UnNeeded_module_B ^
%1

@echo off
for %%F in (%1) do set pyi_output=%%~nxF
set pyi_output=%pyi_output:~0,-3%

mkdir dist\%pyi_output%\windll
mkdir dist\%pyi_output%\lib

move dist\%pyi_output%\*.dll dist\%pyi_output%\windll
move dist\%pyi_output%\*.pyd dist\%pyi_output%\lib

move dist\%pyi_output%\windll\python36.dll dist\%pyi_output%
move dist\%pyi_output%\windll\VCRUNTIME140.dll dist\%pyi_output%
if exist dist\%pyi_output%\windll\pywintypes36.dll (
    move dist\%pyi_output%\windll\pywintypes36.dll dist\%pyi_output%
)

pause

To not mess up your project, create a copy of your code folder and place this compile.bat inside. Then, just drag and drop your main_script.py to compile.bat.

The pause command keeps the console windows open so that you know if the compilation is successful or not.

dragon2fly
  • 2,309
  • 19
  • 23
  • I followed this answer but the app fails to launch after I moved some modules into the specified folder. I added a print in `hooker.py` and noticed the `sys.path` already included the desired paths, meaning the rt-hooker is working as expected but the app wouldn't find modules in those paths, any advice? – nochenon Feb 04 '21 at 06:25
  • Continuing... I'm using pyinstaller 4.2 + python 3.9, here is the output in consoles which you can see the paths were added into `sys.path` already. `['C:\\Users\\nochenon\\dist\\XXX\\base_library.zip', 'C:\\Users\\nochenon\\dist\\XXX', 'C:\\Users\\nochenon\\dist\\XXX\\lib', 'C:\\Users\\nochenon\\dist\\XXX\\dll', 'C:\\Users\\nochenon\\dist\\XXX\\deps']` – nochenon Feb 04 '21 at 06:28
  • @nochenon You have to do a binary search to find which `dll` or `pyd` files must not be moved. Keep your current compiling setting. Move all `dll` file out to see if the app launches as it should. Then move half of the `dll` file back in `windll` to see if the app crashes. If it is, move half of them out and in again until you find a suspected file that should be outside. – dragon2fly Mar 05 '21 at 15:51
  • @nochenon If you have other app running well with this compiling setting, you could also whitelist the same files that are in `windll` and `lib` folder of this app too. This will speed up your process. – dragon2fly Mar 05 '21 at 15:55
  • 1
    I end up modifying the "pyimod03_importers.py" inside the pyinstaller module to get it working: `sys._MEIPASS = pyi_os_path.os_path_join(sys._MEIPASS, "deps")`, I can move everything except python39.dll & base_library.zip into `deps` folder. I know this is not a recommanded way, but the result seems prettry well. Note: if `mulitprocessing` is used, `pyi_rth_multiprocessing.py` should be modified as well. – nochenon Mar 10 '21 at 07:46
-4

another way, new loadmyapp.c:

#include<stdlib.h>
main(int argc,char *argv[]) {
    execv("yourapp/app.exe", argv);
}

gcc -o loadmyapp loadmyapp.c

./loadmyapp

aswjh
  • 1