Map BGR OpenCV Mat to Eigen Tensor

Question

I'm trying to convert an OpenCV 3-channel Mat to a 3D Eigen Tensor.

So far, I can convert 1-channel grayscale Mat by:

    cv::Mat mat = cv::imread("/image/path.png", cv::IMREAD_GRAYSCALE);
    Eigen::MatrixXd myMatrix;
    cv::cv2eigen(mat, myMatrix);

My attempt to convert a BGR mat to a Tensor have been:

    cv::Mat mat = cv::imread("/image/path.png", cv::IMREAD_COLOR);
    Eigen::MatrixXd temp;
    cv::cv2eigen(mat, temp);
    Eigen::Tensor<double, 3> myTensor = Eigen::TensorMap<Eigen::Tensor<double, 3>>(temp.data(), 3, mat.rows, mat.cols);

However, I'm getting the following error :

libc++abi.dylib: terminating with uncaught exception of type cv::Exception: OpenCV(4.1.0) /tmp/opencv-20190505-12101-14vk1fh/opencv-4.1.0/modules/core/src/matrix_wrap.cpp:1195:
error: (-215:Assertion failed) !fixedType() || ((Mat*)obj)->type() == mtype in function 'create'

in the line: cv::cv2eigen(mat, temp);

Any help is appreciated!

Dr Yuan Shenghai · Accepted Answer · 2019-06-14T20:05:49.967

The answer might be disappointing for you.

After going through 12 pages, My conclusion is you have to separate the RGB to individual channel MAT and then convert to eigenmatrix. Or create your own Eigen type and opencv convert function

In OpenCV it is tested like this. It only allows a single channel greyscale image

https://github.com/daviddoria/Examples/blob/master/c%2B%2B/OpenCV/ConvertToEigen/ConvertToEigen.cxx

And in OpenCV it is implemented like this. Which dont give you much option for custom type aka cv::scalar to eigen std::vector

https://github.com/stonier/opencv2/blob/master/modules/core/include/opencv2/core/eigen.hpp

And according to this post,

https://stackoverflow.com/questions/32277887/using-eigen-array-of-arrays-for-rgb-images

I think Eigen was not meant to be used in this way (with vectors as "scalar" types).

they also have the difficulting in dealing with RGB image in eigen.

Take note that Opencv Scalar and eigen Scalar has a different meaning

It is possible to do so if and only if you use your own datatype aka matrix

So you either choose to store the 3 channel info in 3 eigen matrix and you can use default eigen and opencv routing.

Mat src = imread("img.png",CV_LOAD_IMAGE_COLOR); //load  image
Mat bgr[3];   //destination array
split(src,bgr);//split source    
//Note: OpenCV uses BGR color order
imshow("blue.png",bgr[0]); //blue channel
imshow("green.png",bgr[1]); //green channel
imshow("red.png",bgr[2]); //red channel
Eigen::MatrixXd bm,gm,rm;
cv::cv2eigen(bgr[0], bm); 
cv::cv2eigen(bgr[1], gm); 
cv::cv2eigen(bgr[2], rm);

Or you can define your own type and write you own version of the opencv cv2eigen function

custom eigen type follow this. and it wont be pretty

https://eigen.tuxfamily.org/dox/TopicCustomizing_CustomScalar.html
https://eigen.tuxfamily.org/dox/TopicNewExpressionType.html

Rewrite your own cv2eigen_custom function similar to this

https://github.com/stonier/opencv2/blob/master/modules/core/include/opencv2/core/eigen.hpp

So good luck.

Edit

Since you need tensor. forget about cv function

Mat image;
image = imread(argv[1], CV_LOAD_IMAGE_COLOR); 
Tensor<float, 3> t_3d(image.rows, image.cols, 3);

// t_3d(i, j, k) where i is row j is column and k is channel. 
for (int i = 0; i < image.rows; i++) 
  for (int j = 0; j < image.cols; j++) 
  {
       t_3d(i, j, 0) = (float)image.at<cv::Vec3b>(i,j)[0]; 
       t_3d(i, j, 1) = (float)image.at<cv::Vec3b>(i,j)[1];
       t_3d(i, j, 2) = (float)image.at<cv::Vec3b>(i,j)[2];
       //cv ref Mat.at<data_Type>(row_num, col_num)
  }

watch out for i,j as em not sure about the order. I only write the code based on reference. didnt compile for it.

Also watch out for image type to tensor type cast problem. Some times you might not get what you wanted.

this code should in principle solve your problem

Edit number 2

following the example of this

int storage[128];  // 2 x 4 x 2 x 8 = 128
TensorMap<Tensor<int, 4>> t_4d(storage, 2, 4, 2, 8);

Applied to your case is

cv::Mat frame=imread('myimg.ppm');
TensorMap<Tensor<float, 3>> t_3d(frame.data, image.rows, image.cols, 3);

problem is I'm not sure this will work or not. Even it works, you still have to figure out how the inside data is being organized so that you can get the shape correctly. Good luck

Dr Yan Shengai Thank you for your answer. I have notice that you didn't cite the Tensor datatype in your answer. Currently I'm using the approach to slice the channels and converting each Mat into an individual Matrix. I was willing to translate the 3-channel Mat directly into a 3D eigen tensor. — Gal Sight, Jun 12 '19 at 17:24
your error happens at cv::cv2eigen(mat, temp); For eigen tensor type. you can do it directly follow this link https://eigen.tuxfamily.org/dox/unsupported/eigen_tensors.html#title1. To access tensor element you can use t_3d(i, j, k) where i is row j is column and k is channel. t_3d(i, j, 0) = image.at(y,x)[0]; t_3d(i, j, 1) = image.at(y,x)[1] ; t_3d(i, j, 2) = image.at(y,x) [2] . — Dr Yuan Shenghai, Jun 12 '19 at 18:08
Hi, @Dr Yuan Shenghai. Thank you for the update. I have tested here and looks that works properly. Nevertheless, it seems that this double loop is really massive. I'm trying to find a way to load the tensor without copy, like using something like TensorMap. If you find something like this in the future, I'm glad if you can share here. — Gal Sight, Jun 14 '19 at 18:52
@GalSight see edit number 2. Not sure if this is what you needed. Go ahead and check it — Dr Yuan Shenghai, Jun 14 '19 at 20:06

score 1 · Answer 2 · answered Jun 16 '20 at 06:25

Updated answer - OpenCV now has conversion functions for Eigen::Tensor which will solve your problem. I needed this same functionality too so I made a contribution back to the project for everyone to use. See the documentation here:

https://docs.opencv.org/3.4/d0/daf/group__core__eigen.html

Note: if you want RGB order, you will still need to reorder the channels in OpenCV before converting to Eigen::Tensor

Map BGR OpenCV Mat to Eigen Tensor

2 Answers2