0

I am acquiring Point Cloud using Kinect v2 API in Windows 10 64 Bit OS. Below is the code snippet-

depthFrame = multiSourceFrame.DepthFrameReference.AcquireFrame();
colorFrame = multiSourceFrame.ColorFrameReference.AcquireFrame();

if (depthFrame == null || colorFrame == null) return;

depthFrame.CopyFrameDataToArray(depthData);
coordinateMapper.MapDepthFrameToCameraSpace(depthData, cameraSpacePoints);
coordinateMapper.MapDepthFrameToColorSpace(depthData, colorSpacePoints);
colorFrame.CopyConvertedFrameDataToArray(pixels, ColorImageFormat.Rgba);

for (var index = 0; index < depthData.Length; index++)
{
    int u = (int)Math.Floor(colorSpacePoints[index].X);
    int v = (int)Math.Floor(colorSpacePoints[index].Y);

    if (u < 0 || u >= COLOR_FRAME_WIDTH || v < 0 || v >= COLOR_FRAME_HEIGHT) continue;

    int pixelsBaseIndex = v * COLOR_FRAME_WIDTH + u) * COLOR_BYTES_PER_PIXEL;

    float x = cameraSpacePoints[index].X;
    float y = cameraSpacePoints[index].Y;
    float z = cameraSpacePoints[index].Z;

    byte red   = pixels[pixelsBaseIndex + 0];
    byte green = pixels[pixelsBaseIndex + 1];
    byte blue  = pixels[pixelsBaseIndex + 2];
    byte alpha = pixels[pixelsBaseIndex + 3];

    PointXYZRGB point = new PointXYZRGB(); // Color point in 3D
    point.postion(x, y, z);
    point.color(red, green, blue, apha);
}

Please see below a screenshot of the point cloud-

enter image description here

Please look around the orange-colored ball in above picture. Upon close inspection, it is visible that there exists a shift in the point cloud.

I am wondering, why such shift exists and how to remove/minimize it? Any workaround, please.

ravi
  • 6,140
  • 18
  • 77
  • 154

1 Answers1

1

The amount of shift in color overlay and depth map can be due to a number of reasons.

  • Frame acquisition of depth and color frames are not at the same instant (as that is how the _reader_MultiSourceFrameArrived function in kinect SDK works. The timestamps for both cameras are slightly different, hence the slight shift. This is more prominent if you are moving the object in view.
  • The coordinateMapper function in the sdk for mapping the color frame and depth frame uses the camera calibration parameters. The default camera calibration parameters are had coded in the sdk, however there are slight differences in each and every device. You could try to recalibrate the Kinect cameras and use the updated calibration parameters to get the correct overlay of the color and depth maps. Note however, that by siply rplacing the camera calibration parameters in the Kinect Fusion code and recompiling does not work, as the parameters are replaced from the closed-source Kinect fusion dll.So you'll have to write your own code to update each frame at runtime.

Hope this helps.

Atif Anwer
  • 450
  • 5
  • 13
  • Thank you very much. I have following points to say- (1) AFAIK, multisource frame synchronizes all the frames internally. Also, none of the objects were moving while capturing the point cloud. (2) I tried running the code after plugging in another Kinect v2 sensor but I noticed the similar shift in that too. _you'll have to write your own code to update each frame at runtime_ may take too much time which is not worth. Is there any quicker way to minimize this shift? – ravi Feb 01 '18 at 08:37
  • (1) Nope, as per my experience, they are not simultaneous but rather very slightly differently timed. But it is interesting that if none of the objects were moving then the difference is a bit too much than normal for the image you have posted. Right now, no idea why that is so... (2) For the camera calibration, afaik there is no other method because the camera params are overwritten by the KF dll. The params are visible in CameraParameters.cs (see the KF C# demo) and are used in GraphicsCamera.cs file for calibration. You could recompile to see if they work, but afaik they are overwritten. – Atif Anwer Feb 01 '18 at 18:58