3

I've been able to find/create some code that allows me to open the depth and color stream from the OpenNI enabled camera (It is an Orbbec Astra S to be specific). Except unlike with the standard OpenNI Viewer, My stream displays the closest points as darkest and further points as the lighter colors.

How would I be able to change this around so that the points closest to the cameras are shown as lighter (whites) and further away is shown as dark?

#include "stdafx.h"
#include "OpenNI.h" 
#include <iostream>
#include <iomanip>
#include <fstream>
#include <string>
#include <array>
// OpenCV Header
#include <opencv2/core/core.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/calib3d/calib3d.hpp>

using namespace std;
using namespace cv;
using namespace openni;

//Recorder

int main(int argc, char** argv)
{
    Device device;
    VideoStream DepthStream,ColorStream;
    VideoFrameRef DepthFrameRead,ColorFrameRead;

    const char* deviceURI = openni::ANY_DEVICE;
    if (argc > 1)
    {
        deviceURI = argv[1];
    }

    Status result = STATUS_OK;
    result = OpenNI::initialize();
    result = device.open(deviceURI);
    result = DepthStream.create(device, openni::SENSOR_DEPTH);
    result = DepthStream.start();
    result = ColorStream.create(device, openni::SENSOR_COLOR);
    result = ColorStream.start();

       device.setImageRegistrationMode(ImageRegistrationMode::IMAGE_REGISTRATION_DEPTH_TO_COLOR);

    int framenum = 0;
    Mat frame;
    while (true)
    {
        if (DepthStream.readFrame(&DepthFrameRead) == STATUS_OK)
        {
            cv::Mat cDepthImg(DepthFrameRead.getHeight(), DepthFrameRead.getWidth(),
                CV_16UC1, (void*)DepthFrameRead.getData());

            cv::Mat c8BitDepth;
            cDepthImg.convertTo(c8BitDepth, CV_8U, 255.0 / (8000));
            cv::imshow("Orbbec", c8BitDepth);

        }

        if (ColorStream.readFrame(&ColorFrameRead) == STATUS_OK)
        {
            ColorStream.readFrame(&ColorFrameRead);
            const openni::RGB888Pixel* imageBuffer = (const openni::RGB888Pixel*)ColorFrameRead.getData();

            frame.create(ColorFrameRead.getHeight(), ColorFrameRead.getWidth(), CV_8UC3);
            memcpy(frame.data, imageBuffer, 3 * ColorFrameRead.getHeight()*ColorFrameRead.getWidth() * sizeof(uint8_t));

            cv::cvtColor(frame, frame, CV_BGR2RGB); //this will put colors right
            cv::imshow("frame", frame);
            framenum++;
        }

        if (cvWaitKey(30) >= 0)
        {
            break;
        }
    }
    DepthStream.destroy();
    ColorStream.destroy();
    device.close();
    OpenNI::shutdown();
    return 0;
}

-------------------EDIT-------------------

These Images are originally read in as 16bit images, which look like this (note how dark it is):

enter image description here

But after converting to an 8bit image, they look as follows:

enter image description here

James Mallett
  • 827
  • 4
  • 11
  • 27
  • Can you give some example depth images that you captured? – masad Oct 10 '16 at 09:53
  • @masad added pictures above. note that the hand is not black, and is within the working distance. But because it is close it is represented much darker than the wall in the background – James Mallett Oct 10 '16 at 12:44

1 Answers1

4

The image you attached shows that the sensor is capturing the data with directly encoding the distance (in mm) of the object in the depth. This is quite normal for such depth cameras. What we want instead for displaying is higher values for objects closer to the sensor (this is totally opposite to the depth image encoding but useful for displaying).

One can devise a simple depth adjustment function if the operating range of the sensor is known. For Astra S, the operating range is from 0.35m to 2.5m. So what we want now is a function that converts 0.35m -> 2.5m and 2.5m -> 0.35m.

This is pretty straightforward, the only caveat is that you have to take care of the invalid depth pixel (depth == 0) yourself. Here is the code for doing this:

#include "include\opencv\cv.h"
#include "include\opencv\highgui.h"


cv::Mat adjustDepth(const cv::Mat& inImage)
{
    // from https://orbbec3d.com/product-astra/
    // Astra S has a depth in the range 0.35m to 2.5m
    int maxDepth = 2500; 
    int minDepth = 350; // in mm

    cv::Mat retImage = inImage;

    for(int j = 0; j < retImage.rows; j++)
        for(int i = 0; i < retImage.cols; i++)
        {
            if(retImage.at<ushort>(j, i))
                retImage.at<ushort>(j, i) = maxDepth - (retImage.at<ushort>(j, i) - minDepth);
        }

        return retImage;
}


int main ()
{
    cv::Mat inImage;
    inImage = cv::imread("testImage.png", CV_LOAD_IMAGE_UNCHANGED);

    cv::Mat adjustedDepth = adjustDepth(inImage);
    cv::Mat dispImage;
    adjustedDepth.convertTo(dispImage, CV_8UC1, 255.0f/2500.0f);
    cv::imshow(" ", dispImage);

    //cv::imwrite("testImageAdjusted.png", adjustedDepth);
    //cv::imwrite("savedImage.png", dispImage);

    cv::waitKey(0);
    return 0;
}

Here is the output renormalized depth image:

enter image description here

If one wants to further explore what happens in such readjustment function, one can have a look at the histogram for image both before and after applying the adjustment.

Histogram for input depth image (D):

enter image description here

Histogram for negative input depth image (-D):

enter image description here

Histogram for (maxVal-(D-minVal)):

enter image description here

Hope this answers your question.

masad
  • 1,547
  • 1
  • 18
  • 40
  • I can see that the values on the x axis would be the distances, what would the Y-axis be? I'm thinking possibly the number of pixels? or something along those lines – James Mallett Oct 17 '16 at 10:09
  • Its essentially a histogram, so yes the y axis shows the frequency or count of pixels with a certain depth value. Also note that since there are a number of invalid depth pixels (value == 0), the histogram has a peak there. – masad Oct 17 '16 at 18:32