0

My goal is augment my pre-existing image processing pipeline (written in Halide) with OpenCV functions such as NL means denoising. OpenCV functions will not be capable of using Halide's scheduling functionality, so my plan is to realize each Halide Func before each OpenCV stage. The remaining question is how to best convert from a Halide Image (the result of the Func realization) to an OpenCV Mat (as input to an OpenCV function) and from OpenCV Mat to Halide Image when done. My Halide Images are of type float and have 3 channels.

One obvious solution to this is to write functions which copy the data from one data type to the other, but this strikes me as wasteful. Not only will it take precious time to copy over the data, but it will also waste memory since the image will then be stored as two different data types. Is there a way to use pointers or data buffers to simply re-wrap the image data in a new format? Hopefully this process would be reversible so I can go from Halide to OpenCV, and then after the OpenCV function is done back to Halide.

Kantthpel
  • 149
  • 10

2 Answers2

3

buffer_t is gone now, so I should update this answer. The current way to make a buffer that wraps an OpenCV mat (which uses an interleaved storage layout) is:

Halide::Runtime::Buffer<float>::make_interleaved(image.data, image.cols, image.rows, image.channels());

If the OpenCV matrix has padding between the rows, the longer form is:

halide_dimension_t shape[3] = {{0, image.cols, image.step1(1)}, 
                               {0, image.rows, image.step1(0)},
                               {0, image.channels(), 1}};
Halide::Runtime::Buffer<float> buffer(image.data, 3, shape);

a halide_dimension_t is the min coordinate, the extent, and then the stride/step in that dimension.

Andrew Adams
  • 1,396
  • 7
  • 3
2

Yes, you can avoid copying data. I see two possible approaches: either allocate memory yourself and refer to that memory in both an OpenCV Mat instance and a Halide buffer_t structure; or let OpenCV's Mat class allocate the memory and refer to that memory in a buffer_t structure.

For the first approach, you can use a Mat constructor that takes a data pointer:

float* data = new float[3 * width * height];
cv::Mat image(height, width, CV_32FC3, data, AUTO_STEP);

For the second approach, you can use the usual constructor or Mat::create method:

cv::Mat image(height, width, CV_32FC3);

Either way, you can use something like the following code to wrap the memory in a Halide buffer_t structure:

buffer_t buffer;
memset(&buffer, 0, sizeof(buffer));
buffer.host = image.data;
buffer.elem_size = image.elemSize1();
buffer.extent[0] = image.cols;
buffer.extent[1] = image.rows;
buffer.extent[2] = image.channels();
buffer.stride[0] = image.step1(1);
buffer.stride[1] = image.step1(0);
buffer.stride[2] = 1;

Now you should be able to operate on the same memory with both OpenCV and Halide functions.

Eric Stollnitz
  • 36
  • 2
  • Thank you very much for your answer Eric. I've set my code to work in exactly the way you've described, and it works for OpenCV and also for Halide for the most part. The one strange issue is that when I make an image from the buffer_t: Image input(buffer); The output looks wrong. – Kantthpel Oct 11 '16 at 23:54
  • Sorry, looks we can't edit comments. I meant to say that when I save the image as created from the buffer, the output appears to be incorrect. However, when passed through any Func the data appears to be stored correctly, since saving the output of a Func which performs no computation results in the correct output. It appears that the act of passing the Image through the Func is correcting something wrong in the Image created from the buffer – Kantthpel Oct 11 '16 at 23:57
  • An alternative proposal for the strange behavior might be that the save_image halide function is unable to correctly parse the strides defined within the buffer. My reason for thinking this is the case is that when the input data is planar then the image looks correct both before and after being passed through a blank Func. However when the input data is interleaved (the default for OpenCV) then only the image which is passed through the blank Func looks correct. – Kantthpel Oct 12 '16 at 00:25
  • I haven't used Halide's Image class enough to know what's going wrong. As you say, save_image may make assumptions about the storage layout that don't hold in your case. You could try using OpenCV's imwrite function to save an image instead. – Eric Stollnitz Oct 12 '16 at 22:11