How to load an image as an input to an onnx model in C++ using DirectML provider

Question

I have been trying to understand how to load an onnx model in C++ using visual studio and provide input to it and see how and what the output of onnx model, But I dont find any way or how to load an input to onnx model

This is the latest Code:

#include<GLFW/glfw3.h>
#define STB_IMAGE_IMPLEMENTATION
#include "stb_image.h"

#include <iostream>
#include <onnxruntime_cxx_api.h>
#include<dml_provider_factory.h>
#include <vector>

std::vector<float> load_image_and_preprocess(std::string& filename, int& x, int& y) 
{
    std::cout << x<<y<<filename<<std::endl;
    x = 0;
    y = 0;
    std::vector<float> ret;
    int n;
    unsigned char* data = stbi_load(filename.c_str(), &x, &y, &n, 3);
    ret.resize(x * y * 3);
    float* rptr = &ret[0 * x * y], * gptr = &ret[1 * x * y], * bptr = &ret[2 * x * y];
    for (int i = 0; i < x * y; i++) {
        *rptr++ = (float(data[3 * i + 0]) / 255 - 0.485) / 0.229;
        *gptr++ = (float(data[3 * i + 1]) / 255 - 0.456) / 0.224;
        *bptr++ = (float(data[3 * i + 2]) / 255 - 0.406) / 0.225;
    }
    stbi_image_free(data);
    return ret;
}


 
int main()
{ 
    const std::string model_s = "C:/Downloads/resnet18-v1-7.tar/resnet18-v1-7/resnet18-v1-7.onnx";
    std::basic_string<ORTCHAR_T> model = std::basic_string<ORTCHAR_T>(model_s.begin(), model_s.end());
    std::cout << model.c_str();
    // onnxruntime setup
    Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "batch-model-explorer");
    Ort::SessionOptions session_options;
    OrtSessionOptionsAppendExecutionProvider_DML(session_options, 0);
    Ort::Session session = Ort::Session(env, model.c_str(), session_options);

    std::cout << "number of model input:" << session.GetInputCount() << std::endl;
    std::cout << "number of model Output:" << session.GetOutputCount() << std::endl;

    Ort::AllocatorWithDefaultOptions allocator;

    auto val = session.GetInputTypeInfo(0).GetTensorTypeAndShapeInfo().GetShape().size();
    auto val1 = session.GetOutputTypeInfo(0).GetTensorTypeAndShapeInfo().GetShape().size();
    std::cout << "number of model input:" << val << std::endl;
    std::cout << "number of model Output:" << val1 << std::endl;

    auto input_names = session.GetInputNameAllocated(0, allocator);
    auto output_names = session.GetOutputNameAllocated(0, allocator);
    std::cout << "Val:" << input_names << std::endl;
    std::cout << "Val:" << output_names << std::endl;


    std::vector<std::vector<int64_t>> input_node_dims(1);
    int i = 0;
    input_node_dims[0] = session.GetInputTypeInfo(0).GetTensorTypeAndShapeInfo().GetShape();

    for (int j = 0; j < input_node_dims[i].size(); j++)
        printf("Input %d : dim %d=%jd\n", i, j, input_node_dims[i][j]);
    
    int x = 0, y = 0;
    std::string sVal = "C:/Desktop/Image.png";
    std::vector<float> vecVal = load_image_and_preprocess(sVal, x, y);
    std::cout << vecVal.size() << std::endl;
    for (int j = 0; j < vecVal.size(); j++)
        std::cout << vecVal[j] << std::endl;
    
    constexpr size_t input_tensor_size = 224 * 224 * 3;
    auto memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
    
    auto input_tensor = Ort::Value::CreateTensor<float>(memory_info, vecVal.data(), input_tensor_size,input_node_dims.data(), 4);
    return 0;
}

Kindly help me out in understanding how to load and provide input to onnx model

Answer 1

The documentation that accompanies this model is clear about the input expectations:

Input

All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (N x 3 x H x W), where N is the batch size, and H and W are expected to be at least 224. The inference was done using jpeg image.

Preprocessing

The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. The transformation should preferably happen at preprocessing.

You can use the venerable stbi_image to load your file from disk:

#define STB_IMAGE_IMPLEMENTATION
#include "stb_image.h"

std::vector<float> load_image_and_preprocess(const std::string& filename, int& x, int& y) {
  std::vector<float> ret;
  int n;
  unsigned char *data = stbi_load(filename.c_str(), &x, &y, &n, 3);
  ret.resize(x * y * 3);
  float *rptr = &ret[0 * x * y], *gptr = &ret[1 * x * y], *bptr = &ret[2 * x * y];
  for (int i = 0; i < x * y; i++) {
    *rptr++ = (float(data[3 * i + 0])/255 - 0.485) / 0.229;
    *gptr++ = (float(data[3 * i + 1])/255 - 0.456) / 0.224;
    *bptr++ = (float(data[3 * i + 2])/255 - 0.406) / 0.225;
  }
  stbi_image_free(data);
  return ret;
}

This code separates the red, green and blue channels as your model seems to expect. This results in a preprocessed vector<float> you can pass to Ort::Value::CreateTensor. Going from there you should probably crib from the squeezenet example