1

I have been trying to integrate ARToolkit Marker Object tracking into a Tango Application.

So far I have created a build so that a tango app can access and use the ARToolkit Native Library or the ARToolkit Unity wrappers. However, they both seem to require exclusive access to the camera in their default configurations.

How could you feed the same Android video feed to both libraries?

  • Could you create a dummy camera device which doubles out the feed?
  • Could you take the tango feed as normal, and then resend it into ARToolkit with a special VideoConf

[edit]
ARToolkit uses the older Camera1 API, takes a onPreviewFrame() callback and passes that byte[] data to it's own Native Library call, which does the actual work.

Along the lines of the second bullet point, could Tango provide a copy of each frames raw camera data using something like iTangoVideoOverlay .
(ARToolkits NDK functionality seems to expect NV21, but can also accept other formats)

If that data was extractable from tango, I believe the ARToolkit NDK functionality can be used without actually owning the camera.

Jethro
  • 3,029
  • 3
  • 27
  • 56
  • Have managed to get image data from Tango using ITangoVideoOverlay, and send it into the ARToolkit by passing it up to the Activity, and then down through a JNI call (no direct library call was available from C#). I need to work out how to change between the format that tango provides and a format that ARToolkit accepts. – Jethro Feb 01 '17 at 16:34

2 Answers2

1

I am afraid that neither of the method you mentioned would work. Tango has exclusive access to camera and I believe ARToolkit also occupies the camera exclusively through camera2 API. With current TangoSDK, I think the walk-around would be use ARToolkit for camera rendering, and Tango for pose tracking.

However, this could expose a problem for time-stamping, which is Tango and ARToolkit has different timestamps. The solution for this is to take a timestamp offset at the very beginning when application starts, and constantly apply that offset when querying pose from Tango based on timestamp.

xuguo
  • 1,816
  • 1
  • 11
  • 15
  • Thanks! I will research more and have a look at the suggestion of splitting the pose/tracking, and the rendering. I have updated the question with an idea to get the image data to ARToolkit's NDK library, if you can let me know if it sounds feasible or if i'm wasting my time, i'd be very grateful! – Jethro Feb 01 '17 at 09:37
0

This blog shows an example integrating the two.

It also links to example source code, but I haven't tidied it up at all after testing - proceed with caution!

You cannot feed the same camera source to both libraries (first bullet point), but you can forward the camera feed from Tango (ITangoVideoOverlay) into ARToolkit ([AcceptVideoImage][2]) (second bullet point).
This is not ideal, because it is fairly inefficient to send the data to Java from C#. The Phab 2 Pro has to downsample the video X4 to achieve a decent framerate.
A better answer would replace the AndroidJavaClass calls with pipes/sockets. Also there are many little problems - it's a pretty hacky workaround.

Jethro
  • 3,029
  • 3
  • 27
  • 56