Use multiple images for batch inference cppflow C++

Question

I'm trying to use cppflow library in windows 10 x64 machine in VS2019 C++. I want to inference my model for batch of images (vector <cv::Mat> ). I write a simple code as below for single image and it works correctly:

string im_path{ "..." };
string model_path{ "...\\ocr_model" };

cv::Mat tmp, im;

cv::resize(cv::imread(im_path, cv::IMREAD_GRAYSCALE), tmp, cv::Size(127, 25), 0, 0, cv::INTER_CUBIC);

cv::transpose(tmp, im);

int rows = im.rows; int cols = im.cols; int channels = im.channels();

// Put image in tensor 
std::vector<uint8_t> img_data;
    
auto e = std::end(img_data);
img_data.insert(e, im.data, im.data + im.total() * channels);

auto input = cppflow::tensor(img_data, {rows, cols, channels});
       
input = cppflow::cast(input, TF_UINT8, TF_FLOAT);
auto t = input.get_data<float>();
 
input = input / 255.f;
input = cppflow::expand_dims(input, 0);
    
cppflow::model model{ model_path };
auto output = model({ {"serving_default_input:0", input}}, { "StatefulPartitionedCall:0"});

I want to load multiple images (in code below I use a cloned image as second image). here is what I really want to do:

string im_path{ "..." };
string model_path{ "...\\ocr_model" };

cv::Mat tmp, im;
cv::resize(cv::imread(im_path, cv::IMREAD_GRAYSCALE), tmp, cv::Size(127, 25), 0, 0, cv::INTER_CUBIC);

cv::transpose(tmp, im);
    
int rows = im.rows; int cols = im.cols; int channels = im.channels();

// Put image in tensor 
std::vector<uint8_t> img_data; 

auto im_clone = im.clone();
auto e = std::end(img_data);
img_data.insert(e, im.data, im.data + im.total() * channels);
e = std::end(img_data);
img_data.insert(e, im_clone.data, im_clone.data + im_clone.total() * channels);

auto input = cppflow::tensor(img_data, {2, rows, cols, channels});
       
input = cppflow::cast(input, TF_UINT8, TF_FLOAT);
 
input = input / 255.f;
input = cppflow::expand_dims(input, 0);
    
cppflow::model model{ model_path };
auto output = model({ {"serving_default_input:0", input}}, { "StatefulPartitionedCall:0"});

As you see the difference between the codes are img_data preparation and tensor definition but unfortunately, I get this error:

Unhandled exception at 0x00007FFFF4514ED9 in cppflow_Test.exe: Microsoft C++ exception: std::runtime_error at memory location 0x00000031F72FDBD8.

How can I load multiple images (vector< cv::Mat >) to a tensor and use its corresponding outputs? in other words I need a example for batch inference using cppflow library.

score 0 · Answer 1 · answered Apr 25 '22 at 10:23

Try using std::copy to insert multiple copies of the image (or multiple images) into the img_data vector.

int data_size = rows * cols * channels;
std::vector<uint8> img_data; 
for (size_t i = 0; i < batch_size; i++)
{
    std::copy(im.data, im.data + data_size, std::begin(img_data) + i * data_size);
}
auto input = cppflow::tensor(img_data, {batch_size, rows, cols, channels});

score 0 · Answer 2 · answered Aug 09 '22 at 14:52

Use cppflow::concat to concatenate the n tensors you want to send to the model:

auto input1 = cppflow::decode_jpeg(file1);
input1 = cppflow::expand_dims(input1, 0);
auto input2 = cppflow::decode_jpeg(file2);
input2 = cppflow::expand_dims(input2, 0);

cppflow::tensor dim({0});
std::vector<cppflow::tensor> values;
values.push_back(input1);
values.push_back(input2);

auto inputs = cppflow::concat(dim, values);

auto output = model({ {"serving_default_input:0", inputs}}, { "StatefulPartitionedCall:0"});

Use multiple images for batch inference cppflow C++

2 Answers2