I can't solve the problem with getting correct transforms for camera (e.g. positions) when using OpenCV. I created a scene in 3dsmax 2021 , it contains a simple box and a camera. From this camera I make a render at 1000x1000 resolution. From the scene itself I take the positions of the "corners" of the box that are visible in the camera. For these corners in Photoshop I define the position in the rendered picture.
Then I use these values for the method cv2.solvePnP
After that I check these values using the method cv2.projectPoints, and add to the renderer the points defined manually in photoshop (red points) and the points obtained by projection (green). If you look at the picture, the deviations from the original are not significant.
Projection points test image
Then I get the rotation matrix using the cv2.Rodrigues method. And then I multiply it by vector Translation to get the position of the camera. In my scene the camera has a position (XYZ): [-146.505,-156.897,127.539] , and the "found" camera position is [-121.82952111, -131.33087704, 110.92139656]. And this is quite far from the truth.
I use :
3dsmax 2021Python 3.7.6numpy 1.21.6opencv-python 4.4.0.42
My code :
import cv2
import numpy as np
from math import pi,atan2,asin
img = cv2.imread("C:\\Users\\Inu\\Desktop\\BoxRender.jpg")
h, w = img.shape[:2]
imagePoints2D = [
(495, 341),
(636, 390),
(503, 442),
(363, 382),
(375, 604),
(502, 681),
(624, 614)
]
modelPoints3D = [
(12.0976,11.2015,47.5239),
(12.0976,-26.0509,47.5239),
(-21.178,-26.0509,47.5239),
(-21.178,11.2015,47.5239),
(-21.178,11.2015,0),
(-21.178,-26.0509,0),
(12.0976,-26.0509,0)
]
imagePoints2D = np.array(imagePoints2D, dtype="double")
modelPoints3D = np.array(modelPoints3D, dtype="double")
distCoeffs = np.zeros((4,1))
focal_length = w
center = (w/2., h/2.)
cameraMatrix = np.array(
[[focal_length, 0, (w)/2],
[0, focal_length, (h)/2],
[0, 0, 1]], dtype = "double"
)
#solvePnP
success, vectorRotation, vectorTranslation = cv2.solvePnP(modelPoints3D, imagePoints2D, cameraMatrix, distCoeffs, flags=cv2.SOLVEPNP_ITERATIVE)
#Test the solvePnP by projecting the 3D Points to camera
projPoints = cv2.projectPoints(modelPoints3D, vectorRotation, vectorTranslation, cameraMatrix, distCoeffs)[0]
imagePoints2D = np.int32(imagePoints2D).reshape(-1,2)
projPoints = np.int32(projPoints).reshape(-1,2)
for p in imagePoints2D:
cv2.circle(img, ((p[0]), (p[1])), 3, (0,0,255), -1)
for p in projPoints:
cv2.circle(img, ((p[0]), (p[1])), 3, (0,255,0), -1)
rotationMatrix = cv2.Rodrigues(vectorRotation)[0]
cameraPos = -np.matrix(rotationMatrix).T @ np.matrix(vectorTranslation)
print("CameraPos :" + str(cameraPos))
#~ Create projection matrix :
projectionMatrix = cameraMatrix.dot(np.hstack((rotationMatrix, vectorTranslation)))
#~ Get the Euler angles:
angles = cv2.decomposeProjectionMatrix(projectionMatrix)[-1]
print("projectionMatrix :" + str(projectionMatrix))
print("Angles :" + str(angles))
#Errors calculating
error = cv2.norm(imagePoints2D,projPoints,cv2.NORM_L2)/len(projPoints)
print("Reprojection error:", error)
cv2.imshow("Final",img)
cv2.waitKey(0)
cv2.destroyAllWindows()
Console Output :
CameraPos :[[-121.82952111]
[-131.33087704]
[ 110.92139656]]
projectionMatrix :[[ 1.03210519e+03 -3.65562292e+02 -2.26104142e+02 1.02811052e+05]
[-4.38527617e+00 -4.75481465e+00 -1.11801528e+03 1.22853106e+05]
[ 6.14326068e-01 6.46494899e-01 -4.52380180e-01 2.09926434e+02]]
Angles :[[124.9821305]
[-37.9029675]
[-23.2559097]]
Reprojection error: 0.7824607964359516
Can someone help me? I can't figure out where I'm going wrong.
I tried different solutions in the documentation as well as on other sites (e.g. here). I also double-checked the input values (positions of corners in 3d and 2d)
Update 21.03.2021 :
So, I've moved on a little bit. The method cv2.solvePnP works correctly, the problem is in finding the right values, in my case it is in finding the focal length. Since I do the test in 3dsmax and I know the camera parameters (FOV was enough for me here), I can calculate the focal length of the width and height using the formula : focalLengthWidth = ImageWidth / (2 * tan(FOV/2)) , for height the same way. After recalculating the focal length , the cv2.solvePnP method returned vectorRotation and vectorTranslation values , with which I got the camera position close to the original : Camera position in 3dsmax (XYZ): [-146.505,-156.897,127.539] and OpenCV found camera position (XYZ): [-146.18032088,-157.10845474,127.56138865].
It remains to deal with the calculation of the focal length at unknown camera parameters (such as FOV).
My changed code:
import cv2
import numpy as np
from math import pi,atan2,asin
image = cv2.imread("C:\\Users\\Inu\\Desktop\\BoxRender.jpg")
imageHeight, imageWidth = image.shape[:2]
imagePoints2D = [
(495, 341),
(636, 390),
(503, 442),
(363, 382),
(375, 604),
(502, 681),
(624, 614)
]
modelPoints3D = [
(12.0976,11.2015,47.5239),
(12.0976,-26.0509,47.5239),
(-21.178,-26.0509,47.5239),
(-21.178,11.2015,47.5239),
(-21.178,11.2015,0),
(-21.178,-26.0509,0),
(12.0976,-26.0509,0)
]
imagePoints2D = np.array(imagePoints2D, dtype="double")
modelPoints3D = np.array(modelPoints3D, dtype="double")
distCoeffs = np.zeros((4,1))
#~ focalLengthWidth = ImageWidth / (2 * tan(FOV/2))
#~ For test our FOV = 45degrees , like in Current Camera from 3dsmax
FOV = 45.0
focalLength = imageWidth / (2 * np.tan(np.deg2rad(FOV/2)))
print (focalLength)
center = (imageWidth/2., imageHeight/2.)
cameraMatrix = np.array(
[[focalLength, 0, (imageWidth)/2.],
[0, focalLength, (imageHeight)/2.],
[0, 0, 1]], dtype = "double"
)
#solvePnP
success, vectorRotation, vectorTranslation = cv2.solvePnP(modelPoints3D, imagePoints2D, cameraMatrix, distCoeffs, flags=cv2.SOLVEPNP_ITERATIVE)
#Test the solvePnP by projecting the 3D Points to camera
projectionPoints = cv2.projectPoints(modelPoints3D, vectorRotation, vectorTranslation, cameraMatrix, distCoeffs)[0]
imagePoints2D = np.int32(imagePoints2D).reshape(-1,2)
projectionPoints = np.int32(projectionPoints).reshape(-1,2)
for p in imagePoints2D:
cv2.circle(image, ((p[0]), (p[1])), 3, (0,0,255), -1)
for p in projectionPoints:
cv2.circle(image, ((p[0]), (p[1])), 3, (0,255,0), -1)
rotationMatrix = cv2.Rodrigues(vectorRotation)[0]
cameraPos = -np.matrix(rotationMatrix).T @ np.matrix(vectorTranslation)
print("CameraPos :" + str(cameraPos))
#~ Create projection matrix :
projectionMatrix = cameraMatrix.dot(np.hstack((rotationMatrix, vectorTranslation)))
#~ Get the Euler angles:
angles = cv2.decomposeProjectionMatrix(projectionMatrix)[-1]
print("projectionMatrix :" + str(projectionMatrix))
print("Angles :" + str(angles))
#Errors calculating
error = cv2.norm(imagePoints2D,projectionPoints,cv2.NORM_L2)/len(projectionPoints)
print("Reprojection error:", error)
cv2.imshow("Final",image)
cv2.waitKey(0)
cv2.destroyAllWindows()
