How to feed multiple input into opencv dnn

Question

I have a .pb model file. And I loaded the model using opencv's readnetfromtensorflow(). Now I want to use the model to generate predictions. There are 4 types of model input.

Input data

1. 256x256 image
2. 64x64 image
3. 64x64 image
4. array (size is 4)

output data

1. array (size is 2)

To generate the model's predictions, I first had to transform the input into blobFromImages. However, I couldn't use it because each image that needs to be converted has a different size. I also tried inserting each image into a vector in setInput() but it failed. What should I do when there are multiple inputs in this situation?

Here is the code I've tried.

cv::dnn::Net net = cv::dnn::readNetFromTensorflow("model/mfg.pb”);
cv::Mat input_face = cv::dnn::blobFromImage(face, 1, cv::Size(256,256), cv::Scalar(104,177,123), true, false);
cv::Mat input_leye = cv::dnn::blobFromImage(leye, 1, cv::Size(64,64), cv::Scalar(104,177,123), true, false);
cv::Mat input_reye = cv::dnn::blobFromImage(reye, 1, cv::Size(64,64), cv::Scalar(104,177,123), true, false);
cv::Mat input_bbox = (cv::Mat1d(1,4) << face_bbox[0]., face_bbox[1]., face_bbox[2]., face_bbox[3].);

std::vector<cv::Mat> input_image = {input_face, input_leye, input_reye, input_bbox};
net.setInput(input_image);
net.forward();

However, it failed and an error message appeared.

Error libc++abi: terminating with uncaught exception of type cv::Exception: OpenCV(4.5.5) /tmp/opencv-20220714-27380-1eyun69/opencv-4.5.5/modules/core/src/matrix_wrap.cpp:81: error: (-215:Assertion failed) 0 <= i && i < (int)v.size() in function 'getMat_'

Answer 1

You can't set multiple inputs on a cv::dnn::Net.(It's not supported.)

I suggest two alternatives.

1. You do that DNN calculations with Tensorflow and other image/video tasks with OpenCV. You can wrap the data of a tensorflow::Tensor as a cv::Mat. (See Guillaume's answer in this question.)

2. Redesign the functional model into a sequential model.