0

I'm trying to create a script that converts a csv file to a feather file, then compile that script to a stand-alone exe which can be invoked from other applications supplying two parameters - path to a csv file and a path to the output feather file. Environment is Windows 10, Python 311. I've created, and successfully run, a script that does what's required when invoked from the command line like:

python arraowreadfeather.py C:\temp\Feather_test\data.csv C:\temp\Feather_test\data.feather

I've compiled the script to an exe using three tools freely available - py2exe, pyinstaller and auto-py-to-exe.

However, when I compile this script to an exe - either stand-alone or in a single folder - then run it from the command line I get this error:

C:\temp\csvtofeather\dist>arrowreadfeather.exe C:\temp\Feather_test\data.feather Calling from_feather with: C:\temp\Feather_test\data.feather Traceback (most recent call last): File "arrowreadfeather.py", line 28, in <module> File "arrowreadfeather.py", line 22, in main File "arrowreadfeather.py", line 7, in from_feather File "pyarrow\feather.pyc", line 226, in read_feather File "pyarrow\array.pxi", line 830, in pyarrow.lib._PandasConvertible.to_pandas File "pyarrow\table.pxi", line 3990, in pyarrow.lib.Table._to_pandas File "pyarrow\pandas_compat.pyc", line 810, in table_to_blockmanager File "pyarrow\pandas_compat.pyc", line 968, in _reconstruct_index File "pyarrow\pandas-shim.pxi", line 126, in pyarrow.lib._PandasAPIShim.pd.get File "pyarrow\pandas-shim.pxi", line 100, in pyarrow.lib._PandasAPIShim._check_import File "pyarrow\pandas-shim.pxi", line 56, in pyarrow.lib._PandasAPIShim._import_pandas ModuleNotFoundError: No module named 'pyarrow.vendored.version'

I simplified the script just to read an existing feather file and that gives the same error. Here is the simplified script and the error output:

`import sys
import pyarrow.feather as ft

def from_feather(_file: str):
    table = ft.read_feather(_file)
    return table

def main():
    if len(sys.argv) < 2:
        print("Insufficient args for conversion. Specify a source and dest.")
        return None
        
    print("Calling from_feather with: ",sys.argv[1])
    feather_file = sys.argv[1]
    d_table = from_feather(feather_file)

if __name__ == "__main__":
    main()`

C:\Python311\output>arrowreadfeather.exe C:\temp\Feather_test\data.feather Calling from_feather with: C:\temp\Feather_test\data.feather Traceback (most recent call last): File "arrowreadfeather.py", line 23, in File "arrowreadfeather.py", line 19, in main File "arrowreadfeather.py", line 5, in from_feather File "pyarrow\feather.py", line 226, in read_feather File "pyarrow\array.pxi", line 830, in pyarrow.lib._PandasConvertible.to_pandas File "pyarrow\table.pxi", line 3990, in pyarrow.lib.Table._to_pandas File "pyarrow\pandas_compat.py", line 810, in table_to_blockmanager File "pyarrow\pandas_compat.py", line 968, in _reconstruct_index File "pyarrow\pandas-shim.pxi", line 126, in pyarrow.lib._PandasAPIShim.pd.get File "pyarrow\pandas-shim.pxi", line 100, in pyarrow.lib._PandasAPIShim._check_import File "pyarrow\pandas-shim.pxi", line 56, in pyarrow.lib._PandasAPIShim._import_pandas ModuleNotFoundError: No module named 'pyarrow.vendored.version' [20312] Failed to execute script 'arrowreadfeather' due to unhandled exception!

Thanks in advance for any help/pointers.

dqu
  • 1
  • 1

2 Answers2

0

Try creating a file named hook-pyarrow.py and then put this code into it

from PyInstaller.utils.hooks import collect_submodules

hiddenimports = collect_submodules('pyarrow.vendored')
vm.opensource
  • 11
  • 1
  • I don't thing it will entirely solve the problem – vm.opensource Mar 24 '23 at 08:16
  • Please note: I'm not a professional python dev. – vm.opensource Mar 24 '23 at 08:18
  • Thanks for the reply. If I create the file you've mentioned above how/where do I use it in the build process? – dqu Mar 24 '23 at 08:27
  • The code he's posted would usually be put in a 'build.spec' config file that will then set the parameters when you run Pyinstaller. From what I can see, his suggestion will work if you implement the '.spec' file properly. Just pass the 'hiddenimports' variable above to the 'hidden_imports' parameter of the Analysis class (for reference: https://pyinstaller.org/en/stable/spec-files.html#:~:text=The%20spec%20file%20tells%20PyInstaller,contents%20of%20the%20spec%20file.) – Ricardo Udenze Mar 24 '23 at 14:29
0

The information above did solve my problem. Steps taken.

  1. Create a .spec file from the script you want to convert to an exe using this command:

    pyi-makespec --onefile yourscriptfile.py

This creates a .spec file that pyinstaller uses to create the stand-alone exe.

  1. Edit the .spec file in a text editor. In my case I added the following details required to solve my issue:

    from PyInstaller.utils.hooks import collect_submodules allhiddenimports = collect_submodules('pyarrow.vendored')

  2. Edit the field 'hiddenimports' to read as follows:

    hiddenimports=allhiddenimports,

  3. Build the script using the .spec file with the following command: pyinstaller yourspecfile.spec

This did indeed produce an executable that runs without dependency issues.

I don't work on Python that often and needed both those pieces of info. above to get to a working solution. I would like to give joint kudos but...

Thanks.

dqu
  • 1
  • 1