Easiest/most robust to detect shape for OpenCV for Intersection over Union of two objects

Question

I am trying to measure the precision of my marker tracking algorithm via post-processing a video.

My algorithm is: Find a printed planar marker in a Videostream and place a virtual marker at that position. I am working with AR.

Here are two frames of such a video:

Virtual Marker on top of detected marker

Virtual Marker with offset to actual marker

I want to calculate the Intersecion over Union / Jaccard Index of the actual marker and virtual marker. For the first picture it would give me ~98% and the second ~1/5th %. This will give me the quality for my algorithm, how precise and well it works.

I want to get the position and rotation of both markers in each frame with OpenCV and calculate the Jaccard Index. As you can see though, if I directly place a virtual marker on top of the paper marker, I will make it difficult for myself (with OpenCV) to detect them.

My idea is to not place a white marker on top of the actual marker, but place an easily detectable "thing" with a specific color or shape with an offset to the marker, let's say 10cm to the right maybe. Then I subtract the offset. So now, at the best case scenario, the position and rotation of the actual marker and the "thing" with the offset subtracted will be the same.

But what should I use as the easily detectable "thing"? I don't have enough experience with OpenCV to know what (colored?) shape I should use. The augmentation can go in front, behind, left, right... of the actual marker anytime during the video and it should do two things:

1. Not hinder the detection of the actual marker, like currently shown in the pictures

2. Be easily detectable itself

Help would be much appreciated!

Answer 1

Assuming you have enough white background around the visual marker:

You could use colored circles, for example in red, green, blue and black.

Use opencv blob detection [1] to detect all blobs and filter for circular ones:

Look-up average color values for detected blobs and filter for the colors of the circles. Alternatively you could filter the whole image for each color and do blob detection on the filtered images. But this is slower.

Find the centroids (~ center point) of each blob using moments of the blob contours. [2] "Center of multiple blobs in an Image".

Now you have the four pixel positions of your circles. If you know the world coordinates of your light projected circles you can use solvePnP to get a pose from this. Knowing the correct world coordinates is tricky in your case because you project the circle with light on a surface. This involves some 3D geometry. You need to know the transformation from camera coordinate system to pattern projector coordinate system and the projection parameters of your projector.

I guess you send the projected pattern as an image to the projector. I think you can then model the projector as a camera with a certain camera matrix (basically field of view & center point). Naturally you know the pixel coordinates of the projected circles. From this you can compute rays in 3D space (in projector coordinate system). As a starting point see [3]. Intersecting [4] them with the correct surface plane (in projector coordinate system) gives you the 3D coordinates of the projected circle pattern in projector coordinate system. Transform these to camera coordinate system using your known transformation. Now use opencv solvePnP to determine pose of projected light marker.

How to get surface plane? If your setup is static you could use visual marker detection of all recorded images and use mean oder median of marker pose as surface plane. Not sure what this implies for your evaluation though..

[1] https://www.learnopencv.com/blob-detection-using-opencv-python-c/

[2] https://www.learnopencv.com/find-center-of-blob-centroid-using-opencv-cpp-python/

[3] https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html

[4] https://www.cs.princeton.edu/courses/archive/fall00/cs426/lectures/raycast/sld017.htm