ModuleNotFound error in a windows application created with pyinstaller

Question

I am building a PyQt5 project, that executes some database operations, and I am trying to create an executable with pyinstaller at Windows 10.

I have created a conda environment, with the relative libraries and my folder structure is like the following:

application
    |_ __init__.py
    |_ main_designer.py
    |_ main_designer.ui
    |_ db
        |_ __init__.py
        |_ db_construct.py
        |_ model
            |_ __init__.py
            |_ ecj.py      (some problem-specific data-model)

main_designer.py is importing db and db.model, and has the following structure:

import PyQt5.QtWidgets as qtw
from PyQt5.QtCore import QProcess
from PyQt5.uic import loadUi

import sys
import os

# sys.path.append(os.path.abspath('db')) # does not seem to help
import db
import db.model


class MainUI(qtw.QMainWindow):
    def __init__(self, parent=None):
        super(MainUI, self).__init__()
        loadUi('main_designer.ui', self)
        ...

    # class methods


 if __name__ == "__main__":
    # Create the application
    app = qtw.QApplication(sys.argv)
    # Create and show the application's main window
    win = MainUI()
    win.show()
    # Run the application's main loop
    sys.exit(app.exec())

I am creating an executable with pyinstaller trying among others pyinstaller -w -p .\db .\main_designer.py (to include the db in PYTHONPATH) and pyinstaller -w -F .\main_designer.py.

Here is main_designer.spec:

# -*- mode: python ; coding: utf-8 -*-


block_cipher = None


a = Analysis(['main_designer.py'],
             pathex=['.\\db'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             hooksconfig={},
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)

exe = EXE(pyz,
          a.scripts, 
          [],
          exclude_binaries=True,
          name='main_designer',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=False,
          disable_windowed_traceback=False,
          target_arch=None,
          codesign_identity=None,
          entitlements_file=None )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas, 
               strip=False,
               upx=True,
               upx_exclude=[],
               name='main_designer')

With every variant, when I run the executable I get the ModuleNotFound error, when main_designer is including the db module, see below:

The rather strange thing is that when I execute the script from the Python command line, like python main_designer.py, the script runs normally. Is that a virtual environment side-effect?

Answer 1

PyInstaller tries to find every module it needs when it builds the executable, but it's not perfect. In your spec file, note the "hidden imports" list which is currently empty. Try adding each of your project's imports in that list so PyInstaller knows to include them.

hiddenimports=['db', 'db.model', 'someothermodule']

PyInstaller reference to --hidden-import command

Answer 2

A solution to the problem came after some trial and error. Some hints that will hopefully work for others, too:

1. include import errors for local packages with --add-data: For local modules, modules constructed by us, we 'd rather include --add-data "C:/path-to-application/local-module;local-module". Example: for the module db, it had to be --add-data "C:/path-to-application/db;db/".
2. include import errors for global packages as hidden imports: This had to be done for eg. pandas, because after running the .exe, it was throwing a ModuleNotFoundError. After adding --hidden-import "pandas", this stopped.
3. Required project files, like *.ui, *.ini, *.yaml, etc.: Make pyinstaller know about them, with --add-data followed by . Example: --add-data "C:/path-to-application/main_designer.ui;.". The format should be path;., and the . should be added to indicate the exact file.
4. PermissionError: [WinError 5] Access is denied, or similar: pyinstaller, complains that it does not have access to overwrite a folder in the dist directory, where the executable is stored. Simply delete it manually and re-execute the command.
5. If we want to exclude some files, this is not directly possible and someone should rather play with the .spec file. A solution is given here

Here is the pyinstaller command corresponding to the question:

pyinstaller --noconfirm --onedir --windowed 
--hidden-import "pymysql" --hidden-import "lxml" --hidden-import "pandas" --hidden-import "sqlalchemy.ext.declarative" 
--add-data "C:/path-to-application;application/" 
--add-data "C:/path-to-application/db;db/" 
--add-data "C:/path-to-application/db/model;,model/" 
--add-data "C:/path-to-application/config.yaml;." 
--add-data "C:/path-to-application/main_designer.ui;."  

"C:/path-to-application/main_designer.py"

The logic behind --add-data, seems to be:

• for including everything in a directory: --add-data "C:\fullpath-to-dirname;dirname"
• for including a single file: --add-data "C:\fullpath-to-file;.", where we indicate the full path to the file and that we want exactly that file.

auto-py-to-exe, can help a lot with unraveling the exact requirements and giving a command blueprint. Didn't work for me throughout this process, but this might have to do with this rather strange PermissionError, which I chose to try to solve and continue with the command line, after that.