onnxruntime-gpu failing to find onnxruntime_providers_shared.dll when run from a pyinstaller-produced exe file of the project

Question

Short: I run my model in pycharm and it works using the GPU by way of CUDAExecutionProvider. I create an exe file of my project using pyinstaller and it doesn't work anymore.

Long & Detail: In my project I train a tensorflow model and convert it to an onnx file successfully. I then load it like so:

    providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']
    model_session = ort.InferenceSession(model_path, providers=providers)
    prediction = model.run(None, {"input_1": tile_batch})

This works and produces correct predictions.

Then I use pyinstaller to build an exe file of this project (which I can then distribute) like so:

pyinstaller --add-binary %MODEL_PATH%/model.onnx;./resources/ ^
--clean ^
--onefile .\src\<my project>\run.py

I then try to run this exe file on the same input parameters as before and I get this error message:

2022-02-25 17:34:58.4774662 [E:onnxruntime:Default, provider_bridge_ort.cc:937 onnxruntime::ProviderSharedLibrary::Ensure] LoadLibrary failed with error 126 "Das angegebene Modul wurde nicht gefunden." when trying to load "C:\Users\ME
\AppData\Local\Temp\_MEI261442\onnxruntime\capi\onnxruntime_providers_shared.dll"
Traceback (most recent call last):
  File "src\integration_tooling\<myproject>\run.py", line 246, in <module>
    model_session = ort.InferenceSession(model_path, providers=providers)
  File "onnxruntime\capi\onnxruntime_inference_collection.py", line 335, in __init__
  File "onnxruntime\capi\onnxruntime_inference_collection.py", line 379, in _create_inference_session
RuntimeError: D:\a\_work\1\s\onnxruntime\python\onnxruntime_pybind_state.cc:531 onnxruntime::python::CreateExecutionProviderInstance CUDA_PATH is set but CUDA wasn't able to be loaded. Please install the correct version of CUDA and cuDNN as me
ntioned in the GPU requirements page (https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements), make sure they're in the PATH, and that your GPU is supported.

[13704] Failed to execute script 'run_onnx_denoise' due to unhandled exception!

This is unexpected, because before exe-fying it it worked.

The error message about CUDA is misleading, because I have CUDA 11.4 (and the fitting cudnn) installed and the corresponding two system variables are set correctly. Setting them incorrectly leads to the previous (before exe) process not working anymore, as expected. Hence, CUDA is installed correctly for sure.

I suspect that the process of turning the project into an exe file somehow looses the ability to find the dll. According to the documentation of pyinstaller, it is possible to access the folder to where the exe will be unpacked by way of sys._MEIPASS. However, the onnxruntime interface doesn't allow me to meddle with that, or I didn't find out, how.

Any clues, anyone?

score 2 · Answer 1 · answered Feb 25 '22 at 22:20

Solved.

After I wrote the question I realized that the error message was specifically that it couldn't find the dll. Well, turns out the message was spot on!

The pyinstaller missed packaging the two decisive dlls into the single exe file. Using the explicit options to package them like so it works:

´´´ --add-binary venv/Lib/site-packages/onnxruntime/capi/onnxruntime_providers_cuda.dll;./onnxruntime/capi/ ^ --add-binary venv/Lib/site-packages/onnxruntime/capi/onnxruntime_providers_tensorrt.dll;./onnxruntime/capi/ ^ ´´´

In my case, there was one more dll and a pyd missing from this import. I add all these using *.dll and *.pyd — tayyab, Jun 07 '22 at 08:38

onnxruntime-gpu failing to find onnxruntime_providers_shared.dll when run from a pyinstaller-produced exe file of the project

1 Answers1