Converting 16 bit depth frame from Intel Realsense D455 to OpenCV Mat in Android Java

Question

I am trying to convert a DepthFrame object that I have obtained from the Intel Realsense D455 camera to an OpenCV Mat object in Java. I can get the the target depth of a pixel using DepthFrame.getDistance(x,y) but I am trying to get the whole matrix so that I can get the distance values in meters, similar to the sample code in their Github repo, which is in C++.

I can convert any color image obtained from the camera stream (VideoFrame or colored DepthFrame) to a Mat since they are 8 bits per pixel using the following function:

    public static Mat VideoFrame2Mat(final VideoFrame frame) {
        Mat frameMat = new Mat(frame.getHeight(), frame.getWidth(), CV_8UC3);
        final int bufferSize = (int)(frameMat.total() * frameMat.elemSize());
        byte[] dataBuffer = new byte[bufferSize];
        frame.getData(dataBuffer);
        ByteBuffer.wrap(dataBuffer).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer().get(dataBuffer);
        frameMat.put(0,0, dataBuffer);
        return frameMat;
    }

However, the un-colorized DepthFrame values are 16 bits per pixel, and the above code gives an error when the CV_8UC1 is substituted with CV_16UC1. The error arises because in the Java wrapper of the OpenCV function Mat.put(row, col, data[]), there is a type check that allows only 8 bit Mats to be processed:

    // javadoc:Mat::put(row,col,data)
    public int put(int row, int col, byte[] data) {
        int t = type();
        if (data == null || data.length % CvType.channels(t) != 0)
            throw new UnsupportedOperationException(
                    "Provided data element number (" +
                            (data == null ? 0 : data.length) +
                            ") should be multiple of the Mat channels count (" +
                            CvType.channels(t) + ")");
        if (CvType.depth(t) == CvType.CV_8U || CvType.depth(t) == CvType.CV_8S) {
            return nPutB(nativeObj, row, col, data.length, data);
        }
        throw new UnsupportedOperationException("Mat data type is not compatible: " + t);
    }

Therefore I tried to use the constructor of Mat that accepts the array and wrote the following method:

    public static Mat DepthFrame2Mat(final DepthFrame frame) {
        byte[] dataBuffer = new byte[frame.getDataSize()];
        frame.getData(dataBuffer);
        ByteBuffer buffer = ByteBuffer.wrap(dataBuffer).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer().get(dataBuffer);

        Log.d(TAG, String.format("DepthFrame2Mat: w: %s h: %s capacity: %s remaining %s framedatasize: %s databufferlen: %s framedepth: %s type: %s " ,
                frame.getWidth(), frame.getHeight(), buffer.capacity(), buffer.remaining(), frame.getDataSize(), dataBuffer.length, frame.getBitsPerPixel(), frame.getProfile().getFormat()));
        
        return new Mat(frame.getHeight(), frame.getWidth(), CV_16UC1, buffer);
    }

But now I keep getting the error E/cv::error(): OpenCV(4.5.5) Error: Assertion failed (total() == 0 || data != NULL) in Mat, file /build/master_pack-android/opencv/modules/core/src/matrix.cpp, line 428

Using the log command seen in the function, I am checking if the data is empty or null, but it is not. Moreover, the DepthFrame bit depth and type seems to be correct, too:

D/CvHelpers: DepthFrame2Mat: w: 640 h: 480 capacity: 614400 remaining 0 framedatasize: 614400 databufferlen: 614400 framedepth: 16 type: Z16

What could be the reason of this error? Is there a better way to handle this conversion?

Note: I have checked the SO questions such as this and examples on the web, however, all of them are in C++. I don't want to add JNI support just for creating a Mat.

score 2 · Accepted Answer · answered Mar 07 '22 at 12:28

2

Even though not directly an OpenCV API solution, converting the byte array to short array in Java seems to work:

    public static Mat DepthFrame2Mat(final DepthFrame frame) {
        Mat frameMat = new Mat(frame.getHeight(), frame.getWidth(), CV_16UC1);
        final int bufferSize = (int)(frameMat.total() * frameMat.elemSize());
        byte[] dataBuffer = new byte[bufferSize];
        short[] s = new short[dataBuffer.length / 2];
        frame.getData(dataBuffer);
        ByteBuffer.wrap(dataBuffer).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(s);
        frameMat.put(0,0, s);
        return frameMat;
    }

answered Mar 07 '22 at 12:28

mcy

1,209
6
25
37

Sorry, i don't understand much about OpenCv or Realsense, but why converting byte array to short array works? What's the sense of that? I'm facing the same error and i saw this same conversion in some other code. – Juan Ignacio Avendaño Huergo Mar 27 '22 at 18:28
1

@JuanIgnacioAvendañoHuergo The actual underlying type for the `DepthFrame` is `Z16`, which means it has 16 bit `framedepth`, or it is of type `short` (therefore we initialize target `Mat` as `CV_16UC1`). The function `frame.getData` writes this `short` typed array into the byte buffer `dataBuffer` as `byte`. In order to correctly read ("decode") this array back to their original values (of type `short`), we need to buffer it as `short`, using the call `ByteBuffer.asShortBuffer()`. – mcy Mar 28 '22 at 07:16
1

Since a single `short` is twice the size of a `byte`, `short` array will have half the *number of* elements of the corresponding `byte` array, hence the length of `s` is half of the length of `dataBuffer`. – mcy Mar 28 '22 at 07:16

Converting 16 bit depth frame from Intel Realsense D455 to OpenCV Mat in Android Java

1 Answers1