Stack Overflow
Stack Overflow

Questions tagged [tritonserver]

39 questions
0
votes
1 answer

Running Triton Server Inference on AWS GPU Graviton instance

I am currently running a Triton server in production on AWS Cloud using a standard GPU enabled EC2 (very expensive). I have seen these new GPU enabled Graviton instances can be 40% cheaper to run. However, they run on ARM (not AMD). Does this mean I…
jtm123
  • 1
  • 1
0
votes
1 answer

triton inference server: deploy model with input shape BxN config.pbtxt

I have installed triton inference server with docker, docker run --gpus=1 --rm -p8000:8000 -p8001:8001 -p8002:8002 -v /mnt/data/nabil/triton_server/models:/models nvcr.io/nvidia/tritonserver:22.08-py3 tritonserver --model-repository=/models I have…
Zabir Al Nazi
  • 10,298
  • 4
  • 33
  • 60
0
votes
1 answer

Cannot find the definition of a constant

I am trying to add a new accelerator to the Nvidia Triton inference server. One of the last thing I need to do it add a new constant like this one (kOpenVINOExecutionAccelerator) but for some reason I cannot find where it is…
Francois
  • 852
  • 6
  • 17
0
votes
1 answer

Triton Inference Server - tritonserver: not found

I try to run NVIDIA’s Triton Inference Server. I pulled the pre-built container nvcr.io/nvidia/pytorch:22.06-py3 and then run it with the command run --gpus=1 --rm -p8000:8000 -p8001:8001 -p8002:8002 -v/F/models:/models…
Antonina
  • 604
  • 1
  • 5
  • 16
0
votes
0 answers

Triton Inference Server with Python backend Streaming

I am using Triton Inference Server with python backend, at moment send single grpc request does anybody know how we can use the python backend with streaming, because I didn't find any example or anything related to streaming the documentation.
Rizwan Ishaq
  • 71
  • 1
  • 1
  • 4
0
votes
1 answer

pose estimation on Triton inference server

I am struggling with running pose models in NVIDIA Triton inference server. The model (open pose , alpha pose , HRNet ... etc ) load normally but the post processing is the problem
younes
  • 1
  • 1
0
votes
1 answer

faster_rcnn_r50 pretrained converted to ONNX hosted in Triton model server

I went through the mmdetection documentation to convert a pytorch model to onnx here link All installations are correct and i'm using onnxruntime==1.8.1, custom operators for ONNX Runtime MMCV_WITH_OPS. I'm using the…
Nrepesh Joshi
  • 27
  • 1
  • 5
0
votes
0 answers

Cmake on centos/rhel system installs to .../lib64 while on ubuntu it installs to .../lib

I'm trying to compile triton inference server on centos/rhel instead of ubuntu. One problem I encounter is that I'll get the following error for some packages (e.g. protobuf, prometheus-cpp): Could not find a package configuration file provided by…
MaGi
  • 171
  • 1
  • 1
  • 10
0
votes
0 answers

Triton into Gitlab CI

Having problems with implementing triton service into gitlab CI. As I noticed in the triton github https://github.com/triton-inference-server/server, they don't have any exposed port by default in Dockerfile and I'm not really able to access the…
Leemosh
  • 883
  • 6
  • 19
1 2
3