1

I use the OnFrameAvailable callback to sync rgb with depth data. If I just render the rgb point cloud everything is fine.

But if I do some image processing the camera throws the following exceptions:

E/camera-metadata: /home/ubuntu/jobs/redwood_internal/RedwoodInternal/Redwood/common/player-engine/src/camera-metadata.cc:56 RAW failed to match frame

or

E/camera-metadata: /home/ubuntu/jobs/redwood_internal/RedwoodInternal/Redwood/common/player-engine/src/camera-metadata.cc:56 YUV failed to match frame

And TangoImageBuffer has complete garbage values. Sometimes black pixels, or the buffer half-half old and new pixel data.

I tried to solve it by threading. Everytime I got a new point cloud, the extra image processing thread needs about 1 sec cpu time. And it helped a little. After a few seconds the same behavior happend.

The problem is that I can't debug the nativ code properly. The monitoring of android studio shows normal cpu and gpu usage.

I've seen that user guppy had this problem with the Leipniz tango version, but no solution was posted. So I hope maybe someone else has managed this problem? Or has any suggestions?

EDIT

The behaviour disappeared, after using the tango_support library to copy xyz and yuv buffers.

Community
  • 1
  • 1
bashbug
  • 556
  • 4
  • 16
  • This is probably a timing problem. If too much time is being taken in the callback doing image processing then it may be having trouble synchronizing. Try to copy the data out and process it in another thread, allowing other onFrameAvailable callbacks to proceed (and ignore them while the data is processing). – xuguo Feb 19 '16 at 21:21
  • I just did a test with a 1second sleep in the OnFrameAvailable callback, it reproduce the exact same error. I think the best way is to process this in the long running thread (i.e render thread). The video overlay c++ example actually shows this, here's the callback function that copies out data: https://github.com/googlesamples/tango-examples-c/blob/master/video-overlay-jni-example/app/src/main/jni/video_overlay_app.cc#L53 – xuguo Feb 19 '16 at 21:38

1 Answers1

0

The "YUV failed to match frame" is most likely caused by the callback thread is taking too much time to execute. In short, you shouldn't do heavy processing in the OnFrameAvailable callback. This also applies to all other Tango callbacks, i.e pose or depth callback.

The solution to this would be copying out the byte buffer data and process it in another thread, potentially, the render thread. In the tango-example-c video-overlay-jni-example, the application does a memcpy to copy the data from callback thread to the render thread, so the processing of the data would not block the callbacks keep coming. See this line.

xuguo
  • 1,816
  • 1
  • 11
  • 15
  • Thanks for your answer! I already do only copy the buffers within the callbacks. The sync of rgb and xyz is in an extra thread. If I only sync those data buffers to an rgb point cloud and render it, everything is fine. The errors only show up, if I do any other computation in a second thread e.g. removing outliers in the point cloud. – bashbug Feb 19 '16 at 22:00
  • I use the tango_support lib to copy xyz and frame buffers. And everything is smooth now. – bashbug Feb 21 '16 at 17:40
  • Yes, Tango support library would be a good option, it handles both buffer copying and threading. – xuguo Feb 23 '16 at 18:20