Stack Overflow
Stack Overflow

Questions tagged [onnxruntime]

ONNX Runtime is a cross-platform inference and training machine-learning accelerator.

See onnxruntime github project.

292 questions
0
votes
0 answers

Why does my C++ program suddenly release memory

These days, I'm trying to determine whether my C++ method (still) has a memory leak. To this end, I wrote a special main that runs the actual method infinitely over and over and record the memory footprint of the process every minute. One run of the…
Green绿色
  • 1,620
  • 1
  • 16
  • 43
0
votes
1 answer

Convert TensorFlow data to be used by ONNX inference

I'm trying to convert a LSTM model from TensorFlow into ONNX. The code for generating data for TensorFlow model training is as below: def make_dataset(self, data): data = np.array(data, dtype=np.float32) ds =…
user2552845
  • 45
  • 1
  • 5
0
votes
1 answer

onnxruntime-c++ infer time usage is fluctuating when I input single sentence, but input a texts file is ok

I'm trying to deploy a bert model on onnxruntime-c++, but the infer time usage is confusing me. The time usage is longer and fluctuating when input is a single sentence from console real time than input a texts file with a lot of sentences. The…
sheDemonstrate
  • 1
  • 1
0
votes
1 answer

ONNX runtime web, How to invoke operations?

This sounds like something the documentation should cover, however, it doesn't and in the many tutorials I've seen nobody seems to have figured this out. I'm trying to run onnx runtime web with a BERT model exported from hugging face. I do get all…
Juan Alberto López Cavallotti
  • 4,644
  • 3
  • 25
  • 43
0
votes
1 answer

How do you use Pytorch model's function in Onnx to get output instead of model.forward() function

TL;DR: How can I use model.whatever_function(input) instead of model.forward(input) for the onnxruntime? I use CLIP embedding to create embedding for my Image and texts as: Code is from the official git merge ! pip install ftfy regex tqdm ! pip…
Deshwal
  • 3,436
  • 4
  • 35
  • 94
0
votes
1 answer

Expand dimension of "EmguCV.Mat" or "Onnx Tensor"

I am using Onnxruntime in C# for yolov4. Here is the pretrained yolo model: https://github.com/onnx/models/tree/main/vision/object_detection_segmentation/yolov4/model EmguCV is used to get an image, and then preprocess it to suit Yolo's input. This…
mary
  • 23
  • 5
0
votes
1 answer

Onnxruntime NodeJS set intraOpNumThreads and interOpNumThreads by execution mode

I'm using Onnxruntime in NodeJS to execute onnx converted models in cpu backend to run inference. According to the docs, the optional parameters are the following: var options = { /** * */ …
loretoparisi
  • 15,724
  • 11
  • 102
  • 146
0
votes
0 answers

How to use onnxruntime parallel with flask?

Created a server that want to run a session of onnxruntime parallel. First question, will be used multi-threads or multi-processings? Try to use multi-threads, app.run(host='127.0.0.1', port='12345', threaded=True). When run 3 threads that the GPU's…
nistar
  • 13
  • 6
0
votes
0 answers

LSTM ONNX model with batch_size >1 :Non-zero status code returned while running ScatterElements

I have deep learning model with LSTM layer. While performing model inference, I am getting the below error: InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running ScatterElements node.…
code_explorer
  • 41
  • 4
0
votes
2 answers

PyTorch normalization in onnx model

I am doing image classification in pytorch, in that, I used this transforms transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) and completed the training. After, I converted the .pth model file to .onnx file Now, in inference, how…
vishak raj
  • 13
  • 4
0
votes
2 answers

onnx custom op registration

I am looking for a similar feature as keras custom layer in ONNX/Onnxruntime. The way I understand to solve this is to implement a custom operator in onnx for experimentation. The documentation seems to be pointing to implementation in C++ as a…
hmedu
  • 13
  • 5
0
votes
0 answers

Microsoft ML Onnx Runtime cannot find library

I tried to use the library : Package Microsoft.ML.OnnxRuntime -Version 1.10.0 (https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime) for my very first ONNX C++ project. I use Visual Studio 2019. I have no clue how to link "Install-Package…
Adelyn
  • 11
  • 1
0
votes
1 answer

Execute different operations of a NN model graph with different hardware accelerators while using Onnxruntime

I was trying to utilize NNAPI via OnnxRuntime for NN model inferencing on an Android device. Based on the youtube video here: https://www.youtube.com/watch?v=Ij5MoUnLQ0E it is possible to specify the hardware accelerators for operators in the model.…
ashwinjoseph
  • 359
  • 3
  • 12
0
votes
1 answer

ONNXRuntime program crashes after running twice

I am writing a program that should run in VS2013, that receives two images, runs it through an ONNX model with C++, and return the model's output because I am using images, I write the program in VS2019, and create a DLL that will run in VS2013 I…
Tal Levi
  • 59
  • 1
  • 6
0
votes
1 answer

regeneratorRuntime is not defined - onnxruntime-web-bundler

I tried to replicate the example found here: https://github.com/microsoft/onnxruntime-inference-examples/tree/main/js/quick-start_onnxruntime-web-bundler: import * as React from 'react'; import ort from 'onnxruntime-web' import regeneratorRuntime…
Raphael10
  • 2,508
  • 7
  • 22
  • 50
1 2 3
…
19 20