How to convert realsense RGB frame to cv::Mat in ubuntu (or other linux env)?

Question

There is no official SDK for doing so， can someone help？ How to convert realsense RGB frame to cv::Mat in ubuntu (or other linux env)?

Answer 1

Seems to be that you can create 3 cv::Mat to represent each of the component of the image RGB + Depth + IR. Therefore, It should be something like

cv::Mat frameColor = cv::Mat::zeros(resolutionColor.height, resolutionColor.width, CV_8UC3); cv::Mat frameDepth = cv::Mat::zeros(resolutionDepth.height, resolutionDepth.width, CV_32FC1); cv::Mat frameIR = cv::Mat::zeros(resolutionIR.height, resolutionIR.width, CV_8UC1);

Check this forum -> https://software.intel.com/en-us/forums/realsense/topic/538066 and Convert a PXCImage into an OpenCV Mat

Answer 2

If you are using the new librealsense SDK for ubuntu: then it is as easy as to create a new instance of a cv::Mat to the heigth and width of the camera and with the dimensions of the image you acquiring.

cv::Mat vis = cv::Mat(1080,1920,CV_8UC3);

After that the only step left is to assign the data given by the stream to the vis.data property of the matrix

vis.data = (uchar*)reinterpret_cast<const uint8_t *>(dev->get_frame_data(rs::stream::color));`

the (uchar*) cast is to cast the uint8_t format to the native format of the cv::Mat.

Because the Realsense stream provides the color as BGR and opencv uses RGB to represent the image, the last step is to transform the matrix to the correct format:

cv::cvtColor(vis, vis, CV_BGR2RGB);

Answer 3

once you have the sample image

PXCImage::ImageData c_image;
cv::Mat p_image;
PXCCapture::Sample *sample = sm->QuerySample();

            if (sample->color != NULL)
            {

                sample->color->AcquireAccess(PXCImage::ACCESS_READ_WRITE, PXCImage::PIXEL_FORMAT_RGB32, &c_image);
                PXCImage::ImageInfo inputInfo = sample->color->QueryInfo();
                cv::Mat img = cv::Mat(inputInfo.height, inputInfo.width, CV_8UC4, c_image.planes[0], c_image.pitches[0]);
                sample->color->ReleaseAccess(&c_image);


            }

Answer 4

/***
Returns the next frame if next frame is recorded
Returns the previous frame if next frame is not recorded
***/
void Converter::ConvertPXCImageToOpenCVMat(Intel::RealSense::Image *inImg, Intel::RealSense::ImageData data, cv::Mat *outImg) {
    auto cvDataType = 0;
    auto cvDataWidth = 0;

    auto imgInfo = inImg->QueryInfo();

    switch (data.format) {
        /* STREAM_TYPE_COLOR */
    case Intel::RealSense::Image::PIXEL_FORMAT_YUY2: /* YUY2 image  */
    case Intel::RealSense::Image::PIXEL_FORMAT_NV12: /* NV12 image */
        throw; // Not implemented
    case Intel::RealSense::Image::PIXEL_FORMAT_RGB32:  /* BGRA layout on a little-endian machine */
        cvDataType = CV_8UC4;
        cvDataWidth = 4;
        break;
    case Intel::RealSense::Image::PIXEL_FORMAT_RGB24:  /* BGR layout on a little-endian machine */
        cvDataType = CV_8UC3;
        cvDataWidth = 3;
        break;
    case Intel::RealSense::Image::PIXEL_FORMAT_Y8: /* 8-Bit Gray Image, or IR 8-bit */
        cvDataType = CV_8U;
        cvDataWidth = 1;
        break;
        /* STREAM_TYPE_DEPTH */
    case Intel::RealSense::Image::PIXEL_FORMAT_DEPTH:  /* 16-bit unsigned integer with precision mm. */
    case Intel::RealSense::Image::PIXEL_FORMAT_DEPTH_RAW: /* 16-bit unsigned integer with device specific precision (call device->QueryDepthUnit()) */
        cvDataType = CV_16U;
        cvDataWidth = 2;
        break;
    case Intel::RealSense::Image::PIXEL_FORMAT_DEPTH_F32: /* 32-bit float-point with precision mm. */
        cvDataType = CV_32F;
        cvDataWidth = 4;
        break;
        /* STREAM_TYPE_IR */
    case Intel::RealSense::Image::PIXEL_FORMAT_Y16: /* 16-Bit Gray Image */
        cvDataType = CV_16U;
        cvDataWidth = 2;
        break;
    case Intel::RealSense::Image::PIXEL_FORMAT_Y8_IR_RELATIVE: /* Relative IR Image */
        cvDataType = CV_8U;
        cvDataWidth = 1;
        break;
    default: 
        break;
    }

    // suppose that no other planes
    if (data.planes[1] != nullptr) throw; // not implemented
                                          // suppose that no sub pixel padding needed
    if (data.pitches[0] % cvDataWidth != 0) throw; // not implemented

    outImg->create(imgInfo.height, data.pitches[0] / cvDataWidth, cvDataType);

    //memcpy(outImg->data, data.planes[0], imgInfo.height*imgInfo.width*cvDataWidth * sizeof(pxcBYTE));
    memcpy(outImg->data, data.planes[0], imgInfo.height*imgInfo.width*cvDataWidth * sizeof(uint8_t));
}