I try to use onnx runtime in a custom ipywidget but i have a problem with the *wasm files needed by the wasm backend.
I start from the template made by cookie cutter ts and then add the onnx example code provided by onnx :
So here my widget.ts code :
import {
DOMWidgetModel,
DOMWidgetView,
ISerializers,
} from '@jupyter-widgets/base';
import { MODULE_NAME, MODULE_VERSION } from './version';
// Import the CSS
import '../css/widget.css';
import {InferenceSession,Tensor} from "onnxruntime-web";
// use an async context to call onnxruntime functions.
async function create() {
try {
// create a new session and load the specific model.
//
// the model in this example contains a single MatMul node
// it has 2 inputs: 'a'(float32, 3x4) and 'b'(float32, 4x3)
// it has 1 output: 'c'(float32, 3x3)
const session = await InferenceSession.create('./model.onnx'{ executionProviders: ['wasm']});
console.log("session",session)
// prepare inputs. a tensor need its corresponding TypedArray as data
const dataA = Float32Array.from([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]);
const dataB = Float32Array.from([10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120]);
const tensorA = new Tensor('float32', dataA, [3, 4]);
const tensorB = new Tensor('float32', dataB, [4, 3]);
// prepare feeds. use model input names as keys.
const feeds = { a: tensorA, b: tensorB };
// feed inputs and run
const results = await session.run(feeds);
// read from results
const dataC = results.c.data;
console.log("result",dataC)
} catch (e) {
document.write(`failed to inference ONNX model: ${e}.`);
}
}
export class ExampleModel extends DOMWidgetModel {
defaults() {
return {
...super.defaults(),
_model_name: ExampleModel.model_name,
_model_module: ExampleModel.model_module,
_model_module_version: ExampleModel.model_module_version,
_view_name: ExampleModel.view_name,
_view_module: ExampleModel.view_module,
_view_module_version: ExampleModel.view_module_version,
value: 'Hello World',
};
}
static serializers: ISerializers = {
...DOMWidgetModel.serializers,
// Add any extra serializers here
};
static model_name = 'ExampleModel';
static model_module = MODULE_NAME;
static model_module_version = MODULE_VERSION;
static view_name = 'ExampleView'; // Set to null if no view
static view_module = MODULE_NAME; // Set to null if no view
static view_module_version = MODULE_VERSION;
}
export class ExampleView extends DOMWidgetView {
render() {
this.el.classList.add('custom-widget');
this.value_changed();
this.model.on('change:value', this.value_changed, this);
create()
}
value_changed() {
this.el.textContent = this.model.get('value');
}
}
then i add onnxruntime-web to depencies in package.json
"dependencies": {
"@jupyter-widgets/base": "^1.1.10 || ^2 || ^3 || ^4 || ^5 || ^6",
"onnxruntime-web": "^1.8.0"
},
and "copy-webpack-plugin": "^8.1.1", to devDependencies in order to copy the *.wasm file to dist and nbextension directory
I have also modified webpack.config.js in order to copy the file wasm file.
module.exports = [
/**
* Notebook extension
*
* This bundle only contains the part of the JavaScript that is run on load of
* the notebook.
*/
{
entry: './src/extension.ts',
output: {
filename: 'index.js',
path: path.resolve(__dirname, 'custom_onnx', 'nbextension'),
libraryTarget: 'amd',
publicPath: '',
},
module: {
rules: rules
},
devtool: 'source-map',
externals,
resolve,
plugins: [new CopyPlugin({
// Use copy plugin to copy *.wasm to output folder.
patterns: [{ from: 'node_modules/onnxruntime-web/dist/*.wasm', to: '[name][ext]' }]
})]
},
...
then i do :
yarn run build
jupyter nbextension install --sys-prefix --symlink --overwrite --py custom_onnx
jupyter nbextension enable --sys-prefix --py custom_onnx
I got the following files in the nbextension folder of my environnement:
anaconda3/envs/custom_widget/share/jupyter/nbextensions/custom_onnx/extension.js
anaconda3/envs/custom_widget/share/jupyter/nbextensions/custom_onnx/index.js
anaconda3/envs/custom_widget/share/jupyter/nbextensions/custom_onnx/index.js.map
anaconda3/envs/custom_widget/share/jupyter/nbextensions/custom_onnx/ort-wasm.wasm
anaconda3/envs/custom_widget/share/jupyter/nbextensions/custom_onnx/ort-wasm-simd.wasm
anaconda3/envs/custom_widget/share/jupyter/nbextensions/custom_onnx/ort-wasm-simd-threaded.wasm
anaconda3/envs/custom_widget/share/jupyter/nbextensions/custom_onnx/ort-wasm-threaded.wasm
As you can see the wasm files are here.
But when i run the introduction.ipynb notebook with jupyter notebook i got this in the console :
XHRGEThttp://localhost:8888/ort-wasm-simd.wasm
[HTTP/1.1 404 Not Found 9ms]
GET
http://localhost:8888/ort-wasm-simd.wasm
État
404
Not Found
VersionHTTP/1.1
Transfert7,92 Ko (taille 7,66 Ko)
Politique de référentstrict-origin-when-cross-origin
wasm streaming compile failed: TypeError: WebAssembly: Response has unsupported MIME type 'text/html' expected 'application/wasm' ort-wasm.js:39:340
falling back to ArrayBuffer instantiation ort-wasm.js:39:377
XHRGEThttp://localhost:8888/ort-wasm-simd.wasm
[HTTP/1.1 404 Not Found 4ms]
Aborted(both async and sync fetching of the wasm failed) ort-wasm.js:18:81
failed to asynchronously prepare wasm: RuntimeError: Aborted(both async and sync fetching of the wasm failed). Build with -sASSERTIONS for more info. ort-wasm.js:38:357
Aborted(RuntimeError: Aborted(both async and sync fetching of the wasm failed). Build with -sASSERTIONS for more info.) ort-wasm.js:18:81
Cette page est en mode de compatibilité (quirks). La mise en page peut en être affectée. Pour le mode standard, utilisez « <!DOCTYPE html> ». widget.ts:45:17
Error: Promised response from onMessage listener went out of scope 32 background.js:841:170
it's look like the wasm file is to found on http://localhost:8888/
I try many things to get it work and the only thing which work it's to add :
import { env } from "onnxruntime-web";
env.wasm.wasmPaths='./wasm/'
before calling InferenceSession
and put the wasm file 'ort-wasm-simd.wasm" in a wasm directory in the example directory
But i wandering if there is another way to use wasm files in a custom widget without having to ship it with the notebook but in the widget package.
I'm on ubuntu 16.04 and these is my pip list :
anyio 3.6.2
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
asttokens 2.1.0
attrs 22.1.0
Babel 2.11.0
backcall 0.2.0
backports.functools-lru-cache 1.6.4
beautifulsoup4 4.11.1
bleach 5.0.1
brotlipy 0.7.0
certifi 2022.9.24
cffi 1.15.1
charset-normalizer 2.1.1
coverage 6.5.0
cryptography 38.0.3
custom_onnx 0.1.0.dev0 /home/nallezard/dev/custom_onnx
debugpy 1.6.3
decorator 5.1.1
defusedxml 0.7.1
entrypoints 0.4
exceptiongroup 1.0.1
executing 1.2.0
fastjsonschema 2.16.2
flit_core 3.8.0
idna 3.4
importlib-metadata 5.0.0
importlib-resources 5.10.0
iniconfig 1.1.1
ipykernel 6.14.0
ipython 8.4.0
ipython-genutils 0.2.0
ipywidgets 8.0.2
jedi 0.18.1
Jinja2 3.1.2
json5 0.9.5
jsonschema 4.17.0
jupyter_client 7.4.1
jupyter-contrib-core 0.4.0
jupyter_core 5.0.0
jupyter-nbextensions-configurator 0.5.0
jupyter-server 1.23.1
jupyterlab 3.5.0
jupyterlab-pygments 0.2.2
jupyterlab_server 2.16.2
jupyterlab-widgets 3.0.3
MarkupSafe 2.1.1
matplotlib-inline 0.1.6
mistune 2.0.4
nbclassic 0.4.8
nbclient 0.7.0
nbconvert 7.2.4
nbformat 5.7.0
nbval 0.9.6
nest-asyncio 1.5.6
notebook 6.5.2
notebook_shim 0.2.2
packaging 21.3
pandocfilters 1.5.0
parso 0.8.3
pexpect 4.8.0
pickleshare 0.7.5
pip 22.3.1
pkgutil_resolve_name 1.3.10
platformdirs 2.5.2
pluggy 1.0.0
prometheus-client 0.15.0
prompt-toolkit 3.0.32
psutil 5.9.4
ptyprocess 0.7.0
pure-eval 0.2.2
pycparser 2.21
Pygments 2.13.0
pyOpenSSL 22.1.0
pyparsing 3.0.9
pyrsistent 0.19.2
PySocks 1.7.1
pytest 7.2.0
pytest-cov 4.0.0
python-dateutil 2.8.2
pytz 2022.6
PyYAML 6.0
pyzmq 24.0.1
requests 2.28.1
Send2Trash 1.8.0
setuptools 65.5.1
six 1.16.0
sniffio 1.3.0
soupsieve 2.3.2.post1
stack-data 0.6.0
terminado 0.15.0
tinycss2 1.2.1
tomli 2.0.1
tornado 6.2
traitlets 5.5.0
typing_extensions 4.4.0
urllib3 1.26.11
voila 0.4.0
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 1.4.2
websockets 10.4
wheel 0.38.4
widgetsnbextension 4.0.3
zipp 3.10.0
