Ask about kinect fusion for AR

Question

I want to create an AR application using Kinect Fusion.

I want to insert a 3d model into my kinect fusion reconstruction. The expected output is like these two videos:

https://www.youtube.com/watch?v=gChlRebNloA

https://www.youtube.com/watch?v=MvP4cHfUD5g

How to overlay the 3D object onto the 3d reconstruction?

Is there function in the SDK that can be used to achieve my goal?

Thanks

ps: I use C#

Answer 1

You should probably start to explore the KinectFusionExplorer-WPF KinectFusion available on codeplex:

http://kinectforwindows.codeplex.com/SourceControl/latest#v1.x/ToolkitSamples1.8.0

KinectFusionExplorer-WPF has some nice utility methods for matrix transformations, (virtual) camera configuration, etc.

The process for doing the AR augmentation is:

1. viewport: set your viewport using the same resolution as the one for the camera image,
2. video background: retrieve the video image from the Kinect and displays it in the background (either as done in the SDK examples or using a textured quad with the video image as done more traditionally),
3. geometric registration: use WPF3D and set a Camera object using: the intrinsic parameters of the Kinect camera (or the same ones used by KF), the camera pose you get via GetCurrentWorldToVolumeTransform() from your volume,
4. rendering: set any local transformation you want for your model (xaml) and render (using the way it's generally done with WPF3D).