0

I am trying to project points on an image after estimating its pose as shown in this OpenCV tutorial, but for any textured "base image" instead of a chessboard image.

To match the base image with a frame from my webcam, I'm using ORB and FLANN and it is working perfectly. The 2D border around the image is correctly rendered with cv2.findHomography and cv2.perspectiveTransform(border_points, homography_matrix) to map the points and cv2.polylines to draw them.

Now I should be able to draw 3D axes on the image's surface using the homography matrix, but I'm stuck not knowing how I would do it nor finding how to do it anywhere. What am I missing here?

# target_pts and webcam_pts are arrays of 2D points generated by ORB
homography_matrix, _ = cv2.findHomography(target_pts, webcam_pts, cv2.RANSAC, 5.0)

_, rvecs, tvecs, _ = cv2.decomposeHomographyMat(homography_matrix, calibration_matrix)

axis = np.float32([[3,0,0], [0,3,0], [0,0,-3]]).reshape(-1,3)

# TODO: cv2.Rodrigues() for rvecs ?
axis_points, _ = cv2.projectPoints(axis, rvecs[0], tvecs[0], calibration_matrix, calibration_coefficients)

# Draw axis_points on image
# ...
nickh
  • 279
  • 3
  • 12
  • 1
    I see no fundamental difference to the tutorial. show us the results you get? – Christoph Rackwitz Dec 05 '20 at 19:52
  • The matter is that I don't know how to project the 3D points. In the tutorial, there is a function called `cv2.findChessboardCorners` that solves the pose estimation part, while I don't know if `rvecs[0]` and `tvecs[0]` are problematic here. If I try to draw a `cv.line` between two of the `axis_points`, the output is completely nonsensical (most of times off-screen). What results exactly should I share in my answer? – nickh Dec 05 '20 at 20:03
  • you are using `decomposeHomographyMat`. I would suspect the return values to be wrong, and maybe also the arguments you pass to it. – Christoph Rackwitz Dec 05 '20 at 20:46
  • try flipping your arguments to `findHomography` – Christoph Rackwitz Dec 05 '20 at 20:50
  • I followed the documentation for `decomposeHomographyMat`. The main difference between `solvePnP` (used in the tutorial) and `decomposeHomographyMat` is that the returned values are matrices and not vectors, that's why I call `rvecs[0]` and `tvecs[0]`. This might be the problem... Flipping the arguments in `findHomography` does not return any error (because the dimensions match), but since it was working for `cv2.polylines` it should be already correct – nickh Dec 05 '20 at 21:10
  • decomposeHomography returns up to four pairs of r and t, but only one is right. Can you try all pairs to verify that one of them is right? – Micka Dec 05 '20 at 23:04
  • Yes, I can try all four, but why such function would return 3 "wrong" values randomly? It should be consistent for every image. My intuition was that by combining all four values, the "correct" `rvec` and `tvec` can be evaluated in some way – nickh Dec 06 '20 at 00:20
  • UPDATE: I tried all 16 combinations and none worked. The documentation suggests that the returning vectors contain "up to four _possible_ solutions", but no criteria is informed. I feel like I'm missing something here... – nickh Dec 06 '20 at 01:00

0 Answers0