Stack Overflow
Stack Overflow

Questions tagged [tensorrt]

questions pertaining to NVIDIA's TensorRT system, used for Deep Learning Inference Optimization on GPUs

TensorRT is a system provided by NVIDIA to optimize a trained Deep Learning model, produced from one of a variety of different training frameworks, for optimized inference execution on GPUs. TensorRT can optimize both the latency of execution, as well as the throughput (inferences/sec) of a trained model.

349 questions
25
votes
3 answers

Could not load dynamic library 'libnvinfer.so.7'

I know that this question has been asked a lot, but none of the suggestions seem to work, probably since my setup is somewhat different: Ubuntu 22.04 python 3.10.8 tensorflow 2.11.0 cudatoolkit 11.2.2 cudnn …
Daniel von Eschwege
  • 481
  • 1
  • 4
  • 10
18
votes
3 answers

Deploy Semantic Segmentation Network (U-Net) with TensorRT (no upsampling support)

I am trying to deploy a trained U-Net with TensorRT. The model was trained using Keras (with Tensorflow as backend). The code is very similar to this one: https://github.com/zhixuhao/unet/blob/master/model.py When I converted the model to UFF…
Yayuchen
  • 301
  • 3
  • 6
12
votes
5 answers

ImportError: No module named 'tensorrt'

I installed TensorRT on my VM using the Debian Installation. If I run "dpkg -l | grep TensorRT" I get the expected result: ii graphsurgeon-tf 5.0.2-1+cuda10.0 amd64 …
K41F4r
  • 1,443
  • 1
  • 16
  • 36
10
votes
3 answers

Installing TensorRT- facing issue with libnvinfer7 library (Cuda 10.2) on ubuntu 18.04

I was trying to install tensorRT 7.0 in ubuntu 18.4 (nv-tensorrt-repo-ubuntu1804-cuda10.2-trt7.0.0.11-ga-20191216_1-1_amd64.deb) debian. Followed the documentation…
Amarnath R
  • 973
  • 3
  • 14
  • 33
10
votes
2 answers

Fastest way to run recurrent neural network (inference) on mobile device

What I have: A trained recurrent neural network in Tensorflow. What I want: A mobile application that can run this network as fast as possible (inference mode only, no training). I believe there are multiple ways how I can accomplish my goal, but…
user667804
  • 740
  • 6
  • 25
9
votes
2 answers

Run Tensorflow with NVIDIA TensorRT Inference Engine

I would like to use NVIDIA TensorRT to run my Tensorflow models. Currenly, TensorRT supports Caffe prototxt network descriptor files. I was not able to find source code to convert Tensorflow models to Caffe models. Are there any workarounds?
Evi
  • 401
  • 1
  • 5
  • 6
8
votes
2 answers

Inference with TensorRT .engine file on python

I used Nvidia's Transfer Learning Toolkit(TLT) to train and then used the tlt-converter to convert the .etlt model into an .engine file. I want to use this .engine file for inference in python. But since I trained using TLT I dont have any frozen…
Sharan
  • 691
  • 1
  • 7
  • 16
7
votes
1 answer

How to use Triton server "ensemble model" with 1:N input/output to create patches from large image?

I am trying to feed a very large image into Triton server. I need to divide the input image into patches and feed the patches one by one into a tensorflow model. The image has a variable size, so the number of patches N is variable for each call. I…
Stiefel
  • 2,677
  • 3
  • 31
  • 42
7
votes
3 answers

get "LogicError: explicit_context_dependent failed: invalid device context - no currently active context? " when running tensorRT in ROS

I have an inference code in TensorRT(with python). I want to run this code in ROS but I get the below error when trying to allocate buffer: LogicError: explicit_context_dependent failed: invalid device context - no currently active context? The code…
Mahsa
  • 466
  • 2
  • 7
  • 26
7
votes
1 answer

TF-TRT vs UFF-TensorRT

I found that we can optimize the Tensorflow model in several ways. If I am mistaken, please tell me. 1- Using TF-TRT, This API developer by tensorflow and integreted TensoRT to Tensorflow and this API called as : from…
DeeeepNet
  • 347
  • 3
  • 11
7
votes
0 answers

Importing caffe's PriorBox into TensorRT

We have a Caffe model that contains: layer { name: "foo" type: "PriorBox" prior_box_param { # ERROR HERE # whatever } # etc } Now, following the code in sampleMNIST I try to import my model into TensorRT but get an…
Ken Y-N
  • 14,644
  • 21
  • 71
  • 114
6
votes
2 answers

Jetson NX optimize tensorflow model using TensorRT

I am trying to speed up the segmentation model(unet-mobilenet-512x512). I converted my tensorflow model to tensorRT with FP16 precision mode. And the speed is lower than I expected. Before the optimization i had 7FPS on inference with .pb frozen…
Pavlo Sharhan
  • 174
  • 2
  • 10
6
votes
3 answers

Darknet model to onnx

I am currently working with Darknet on Yolov4, with 1 class. I need to export those weights to onnx format, for tensorRT inference. I've tried multiple technics, using ultralytics to convert or going from tensorflow to onnx. But none seems to work.…
remc
  • 161
  • 1
  • 2
  • 10
6
votes
1 answer

What is the difference between TensorRT GA and RC?

I just wanted to download TensorRT but I saw there are two different versions GA and RC. What is the differences between them and what version should I choose as I use a Windows 10 machine with Geforce GTX 960M graphic card?
Hasani
  • 3,543
  • 14
  • 65
  • 125
6
votes
1 answer

How to use only one GPU for tensorflow session?

I have two GPUs. My program uses TensorRT and Tensorflow. When I run only TensorRT part, it is fine. When I run together with Tensorflow part, I have error as [TensorRT] ERROR: engine.cpp (370) - Cuda Error in ~ExecutionContext: 77 (an illegal…
batuman
  • 7,066
  • 26
  • 107
  • 229
1
2 3
…
23 24